Repository: psf/requests

Issue #6586: The Content-Length header for string `data` counts Unicode characters in the string when it should count encoded bytes
Created: 2023-11-27T22:31:09Z
A call like this:
response = requests.post("https://example.com", data="👍👎")
auto sets the Content-Length header in the request to 2 when it should be 8.
I hit this issue was making a request with a JSON body to a service I own (running behind AWS API Gateway) and having the service complain that there was no closing brace } in the JSON body. I was passing the JSON body into requests as a string to the data argument. It turns out that API Gateway ignores any body bytes beyond the Content-Length in the request. Turning up detailed logging on API Gateway, I can see the request headers and realized the value in the Content-Length header didn't match the number of bytes in the body.
A quick workaround is to encode the string into bytes before passing it into Requests.
This produces a Content-Length header with the correct value of 8:
response = requests.post("https://example.com", data="👍👎".encode("utf-8"))
Expected Result
On a server receiving a POST from Requests, I expect the Content-Length header value to match the number of bytes in the body of the request. See RFC 9110.
Actual Result
In the specific case where Request's data argument is set as a string containing characters which encode into multi-byte UTF-8, the value in the Content-Length header is incorrect. Requests appears to be counting the number of Unicode characters in the string instead of the number of bytes that will be sent to the server.
Reproduction Steps
>>> import requests
>>> thumbs_up_down = "👍👎"
>>> len(thumbs_up_down)
2
>>> len(thumbs_up_down.encode())
8
>>> pending_request = requests.Request("POST", "https://example.com", data=thumbs_up_down)
>>> prepared_request = pending_request.prepare()
>>> prepared_request.headers
{'Content-Length': '2'}
I opened a pull request, #6587, that adds a failing unit test that demonstrates this problem.
System Information
$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.16"
  },
  "platform": {
    "release": "23.1.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "1010111f"
  },
  "urllib3": {
    "version": "2.1.0"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Duplicate issue #6763: Body with Special Characters Gets Cut
Created: 2024-07-08T14:33:37Z
When sending a request with special characters using the requests module, the request gets cut and is not sent fully.
This seems to be caused by the requests module calculating the length of the original string, but once the request arrives at the urllib3 module, the urllib3 module encodes the request and calculates the content length again. Unfortunately, because the request content length was already calculated and included in the headers dictionary, it gets overwritten.
Expected Result
The request should be sent fully, including all special characters.
Actual Result
The request gets cut off, and not all data is sent.
Reproduction Steps
import requests

### Note that the x are special characters ###
response = requests.post(url='http://127.0.0.1', data="""{"test": "××××"}""")
print(response.text)
Example of the actual request:
POST / HTTP/1.1
Host: 127.0.0.1
User-Agent: python-requests/2.31.0
Accept-Encoding: gzip, deflate, br
Accept: */*
Connection: keep-alive
Content-Length: 16

{"test": "×××

As it looks the overwrite happens here:
urllib3 connection.py#L396
System Information
$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "3.3.2"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "3.7"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.11.9"
  },
  "platform": {
    "release": "23.5.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.31.0"
  },
  "system_ssl": {
    "version": "300000d0"
  },
  "urllib3": {
    "version": "2.0.7"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

Closing comment (2024-07-08T18:29:33Z): This is a duplicate of #6586. Please either upgrade to Requests>=2.32.0 or downgrade to urllib3<2. This is a behavioral change in the major version bump of urllib3.