Open
Description
Describe the bug
hi，friends, i have tested a case for 2 weeks, and the python server may had a memory leak ?
By "mprof" tool, the memory always increased near "async with session.post"
i don't know how to analysis the cause of the problem, any advice will be appreciate!!!
the RES memory data by every minute is here
23208
23324
23424
23448
23496
23524
23552
23572
23620
23644
23676
23692
23692
23724
23764
23788
23804
23816
23836
23860
23872
23892
23904
23928
23940
23960
23980
23992
24004
24028
24048
24060
24080
24092
24112
24124
24144
24164
24176
24196
24208
24232
24244
24264
24284
24296
24316
24328
24348
24360
24384
24404
24416
24436
24448
24468
24480
24500
24512
24532
24544
24568
24580
24600
24612
24636
24656
To Reproduce
client code request.py
import aiohttp
import asyncio
import json
import logging
import time
from logging.handlers import RotatingFileHandler
handler = RotatingFileHandler(
filename='app.log',
mode='a',
maxBytes=500 * 1024 * 1024,
backupCount=3,
encoding='utf-8'
)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
logger.setLevel(logging.INFO)
logger.addHandler(handler)
async def post_task(session, url, headers=None, json_param=None):
try:
logger.info(f"Request start")
async with session.post(url, headers=headers, data=json.dumps(json_param)) as resp:
if resp.status != 200:
logger.error(f"Request failed with status {resp.status}")
resp.release()
return None, time.time(), -1
reply = await resp.read()
end_t = time.time()
return reply, end_t, 0
except Exception as e:
logger.error(f"post_task failed: {e}")
return None, time.time(), -1
async def request():
connector = aiohttp.TCPConnector(limit=10)
timeout = aiohttp.ClientTimeout(total=15)
async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:
tasks = [
post_task(session, "http://xx/test_memory/v1/text", headers={"Content-Type": "application/json"}, json_param={"key": "value"}),
post_task(session, "http://xx/test_memory/v1/subjectRecognition", headers={"Content-Type": "application/json"}, json_param={"key": "value"}),
]
results = await asyncio.gather(*tasks, return_exceptions=True)
for result in results:
if isinstance(result, Exception):
logger.error(f"Task failed with exception: {result}")
else:
logger.info(f"Task succeeded: {result}")
async def main():
test = 0
while test <1:
await request()
await asyncio.sleep(0.1)
asyncio.run(main())
then execute "nohup python3 request.py >/dev/null 2>&1 &"
server code fake_server.py
from fastapi import FastAPI, File, UploadFile
from fastapi.middleware.cors import CORSMiddleware
import uvicorn
from pydantic import BaseModel
from fastapi import Request
import logging
import time
from logging.handlers import RotatingFileHandler
handler = RotatingFileHandler(
filename='app.log',
mode='a',
maxBytes=5 * 1024 * 1024,
backupCount=3,
encoding='utf-8'
)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
logger = logging.getLogger('my_logger')
logger.setLevel(logging.INFO)
logger.addHandler(handler)
app = FastAPI(title="Multi-Service API Server")
app.add_middleware(
CORSMiddleware,
allow_origins=["*"],
allow_methods=["*"],
allow_headers=["*"],
)
class EducationRequest(BaseModel):
data: str
model_type: str
version: str
type: str
request_id: str
return_probability: int
@app.post("/test_memory/v1/text")
async def ocr_text(request: Request):
#print("request ocr text")
logger.info(f"request ocr text")
try:
return {
"status": 200,
"message": "ocr_text",
"data": "ocr_text",
}
except Exception as e:
return {"status": "error", "message": str(e)}
@app.post("/test_memory/v1/subjectRecognition")
async def subjectRecognition(request: Request):
logger.info(f"request subjectRecognition")
try:
return {
"code": 0,
"message": "subjectRecognition",
"data": "subjectRecognition",
}
except Exception as e:
return {"status": "error", "message": str(e)}
if __name__ == "__main__":
uvicorn.run(app="fake_server:app", host="0.0.0.0", port=8888,workers=6)
then execute "nohup python3 fake_server.py >/dev/null 2>&1 &"
Expected behavior
The memory is in a relatively stable state，not increases so obviously.
Logs/tracebacks
nillPython Version
Python 3.10.0aiohttp Version
$ python -m pip show aiohttp
Name: aiohttp
Version: 3.8.1
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author:
Author-email:
License: Apache 2
Location: /usr/local/lib/python3.10/site-packages
Requires: aiosignal, async-timeout, attrs, charset-normalizer, frozenlist, multidict, yarl
Required-by: oppo-dapr-python-sdkmultidict Version
$ python -m pip show multidict
Name: multidict
Version: 6.1.0
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache 2
Location: /usr/local/lib/python3.10/site-packages
Requires: typing-extensions
Required-by: aiohttp, sanic, yarlpropcache Version
$ python -m pip show propcache
Name: propcache
Version: 0.2.1
Summary: Accelerated property cache
Home-page: https://github.com/aio-libs/propcache
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache-2.0
Location: /usr/local/lib/python3.10/site-packages
Requires:
Required-by: yarlyarl Version
$ python -m pip show yarl
Name: yarl
Version: 1.18.3
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl
Author: Andrew Svetlov
Author-email: [email protected]
License: Apache-2.0
Location: /usr/local/lib/python3.10/site-packages
Requires: idna, multidict, propcache
Required-by: aiohttpOS
linux
Related component
Server
Additional context
No response
Code of Conduct
- I agree to follow the aio-libs Code of Conduct