CURL doesn't send POSTFIELDS correctly to some servers

Question

I've tried searching, the other answers didn't help me.

I'm trying to send POST data via cURL, but it's only working on some servers. What gives?

I've tried:

if (!empty($data) && $usePost) {
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
}

$data contains an array with the key of products and the value of a JSON string.

I also tried explicitly using the query:

if (!empty($data) && $usePost) {
    curl_setopt($ch, CURLOPT_POST, true);
    curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($data));
}

But still, $_POST on the receiving server gives me NULL.

Any idea what to do?

Here's the whole method:

public static function execCommand($command, $ch,$data=array(),$cookie_file='genCookie.txt', $usePost = false) {
    $url = $command;
    if (!empty($data) && !$usePost)
        $url .= '?' . http_build_query($data);
    elseif (isset($data['sessionid'])) {
        $url .= '?sessionid=' . $data['sessionid'];
        unset($data['sessionid']);
    }

    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
    curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; rv:21.0) Gecko/20100101 Firefox/21.0');
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);

    if (!empty($data) && $usePost) {
        curl_setopt($ch, CURLOPT_POST, true);
        curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
    }

    return curl_exec($ch);
}

I tried echoing $_REQUEST, so it's not in $_GET by mistake. I also tried file_get_contents('php://input'), but still nothing.

Both the sending and receiving ends are on Apache servers.

Bigger thing is, it's working on most servers, but only some are ignoring it. Is there a safer, more cross platform way to do this?

Updated. Both sides are Apache, and the code is in the original question — casraf, Apr 30 '14 at 07:13
when it happens? When you change the client (the server where there is this code) or when you change the endpoint? (the server that receive the post)? — Luca Rainone, May 07 '14 at 07:41
Enable `CURLOPT_VERBOSE` to see the request and response headers — FuzzyTree, May 09 '14 at 21:42
Can you share any information on the number of servers you are attempting and if there is anything different about the servers that are failing? — smcjones, May 11 '14 at 03:41
Set the appropriate headers. Maybe it is a CORS request. http://stackoverflow.com/questions/12173990/how-can-you-debug-a-cors-request-with-curl — GuyT, May 13 '14 at 07:41
Please add how you are reciving/evaluating the $_POST on client side. — tswaehn, May 13 '14 at 20:54

score 4 · Answer 1 · answered May 07 '14 at 09:03

I find it difficult to believe that Curl is not sending "post" fields to some servers (the code you shared looks OK). I would rather assume that there is something wrong with how you handle POST on those servers or you do something wrong elsewhere.

You can troubleshoot with tcpdump

$ sudo tcpdump -nl -w - -s0 -A -c 500 tcp port 80 | strings

Now if you run an example code like:

<?php

class Curl {

    protected static function getCurlHandler( $url, $data = false, $headers = false ) {
        $ch =  curl_init(); 

        curl_setopt( $ch, CURLOPT_URL, $url );
        curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1 );
        curl_setopt( $ch, CURLOPT_HEADER, 0 );

        if( $headers ) {
            curl_setopt($ch, CURLINFO_HEADER_OUT, true);
            curl_setopt( $ch, CURLOPT_HTTPHEADER, $headers );
        }

        if( $data ) {
            curl_setopt( $ch, CURLOPT_POSTFIELDS, http_build_query( $data ) );
        }

        return $ch;
    }

    protected static function exec( $ch ) {
        $output = curl_exec( $ch );

        // handle errors

        curl_close( $ch );
        return $output;
    }

    public static function get( $url, $headers = array() ) {
        $ch = self::getCurlHandler( $url, false, $headers );

        return self::exec( $ch );
    }

    public static function post( $url, $data = array(), $headers = array() ) {
        $ch = self::getCurlHandler( $url, $data, $headers );
        curl_setopt( $ch, CURLOPT_POST, 1 );

        return self::exec( $ch );
    }

    public static function put( $url, $data = array(), $headers = array() ) {
        $ch = self::getCurlHandler( $url, $data, $headers );
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PUT");

        return self::exec( $ch );
    }

    public static function delete( $url, $data = array(), $headers = array() ) {
        $ch = self::getCurlHandler( $url, $data, $headers );
        curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");

        return self::exec( $ch );
    }

}


Curl::post( "http://www.google.com", array('q' => 'hello world'));

You should find something like this in the TCPDUMP output

POST / HTTP/1.1
Host: www.google.com
Accept: */*
Content-Length: 13
Content-Type: application/x-www-form-urlencoded
q=hello+world

The last line is the most interesting bit which clearly shows that CURL sent a POST request with the required data.

score 1 · Answer 2 · answered May 11 '14 at 03:37

Your alternative is to use wget which essentially downloads a file from a location. It is probably going to be a bit slower as you need to then open the file you download and delete the space. I am not confident it scales well and it's not going to be good for POST requests. cURL works using port 80, which is the basic browser port. That means that if you can access it via a browser, cURL will work.

Through the limited information you gave I can provide some troubleshooting tips.

In this instance I am calling your PHP server that initiates the request the client and the server you are sending information to the server.

The first thing you should do is make sure that this is working on your server. Check your source code.

Step 1. Try to submit a form and make sure $_POST returns information. If it does, you can move on. If it doesn't you have a very, very strange bug. I would be interested to know if you are using different OSes on these different servers.

Step 2. Check your server sessionid. Make sure your client is expecting them to come via GET. I know this may seem obvious, but many times the obvious is the source of failure. What does your server do if your sessionid fails? Make sure your processing script returns something like:

if(session_is_registered($_GET['sessionid']) {
    echo "This is a registered session.";
    $_POST['debugSessionExists'] = true;
} else {
    echo "This sesion does not exist.";
    $_POST['debugSessionExists'] = false;
}

You should be able to spit out return data on your client too thanks to the echo.

Step 3. Do the same thing for all of your $_POST variables and echo a return.

foreach($_POST as $key=>$value) {
    echo "{$key} exists and has the value of $value.<BR />";
}
echo "Done checking $_POSTs.";

The reason I advise this is because I am not sure how you have been checking your $_POST values to date and it doesn't hurt to see if your cURL response is getting information you are not seeing. This means that the way you are storing your $_POST values is just not transferring on some of your servers and that is your point of failure.

I will be interested to find out what is returned via these results.

score 0 · Answer 3 · edited Aug 27 '22 at 13:15

As mentioned earlier, you might need to do tcpdump to see exactly how the data is going to the server.

You might be running into the 'Expect' header with curl which can be disabled:

http://the-stickman.com/web-development/php-and-curl-disabling-100-continue-header/

By disabling the Expect header, there is a perf. advantage of avoiding additional round-trips between the client and the server albeit at the cost of bypassing additional checks on the server side (e.g. enforcing maximum POST body size etc...).

score 0 · Answer 4 · answered May 14 '14 at 04:34

0

Try debugging with http://req.uest.info/

Also have you checked the logs of your originating server?

answered May 14 '14 at 04:34

Chris Gunawardena

6,246
1
29
45

score 0 · Answer 5 · answered May 14 '14 at 06:49

Bigger thing is, it's working on most servers, but only some are ignoring it.

The POST is sent ok, but the server differ. For starters, compare the PHP versions.

For example; since PHP 4.2.0 register_globals is off, which could give similar NULL results if the receiving end reuses variable names like $foo VS $_POST['foo'].

Is there a safer, more cross platform way to do this?

No, a POST is a good way to send data. But keep in mind that data is sent plaintext over the wire unless you use encryption or https.

score 0 · Accepted Answer · answered May 14 '14 at 14:47

Turns out the problem, was the SSL on the receiving server. The request was being made to HTTP instead of HTTPS, and instead of giving an error or redirection, it simply dropped the POST on the way (maybe it redirected to HTTPS but failed to pass the POST along with it?)

CURL doesn't send POSTFIELDS correctly to some servers

6 Answers6