How to use boost.python to call c++ dynamic library in linux

Preface

Recently I started using robot framework to test c++ dynamic libraries. robot framework runs on windows and c++ dynamic libraries run on a remote linux host. The test method is to let the robot framework execute the python script on the remote machine through the SSHLIbrary library, and the python script calls the C++ dynamic library. So now we need to solve the problem of how to let Python call the C++ dynamic library.

Two ways to call C++ dynamic library in Python

After searching online and consulting colleagues, I found two methods: the first is to encapsulate the C++ dynamic library into a C interface and let Python call the C language interface. Since Python can only call C interfaces and cannot directly call C++ interfaces, a layer of encapsulation is required. Encapsulation method: Use the extern "C" declaration method to encapsulate a layer of C language interface on top of the C++ interface. After trying this method, we found that pure C calls are feasible, but Python calls are not feasible. The reasons are explained in detail below. The second method is to use the C++ boost library to generate an interface for Python to call. It has been tested and is feasible, but the process is very tortuous. The following will explain in detail the problems encountered and their solutions.

Understanding the nature of extern "C"

Before describing the first method, let's briefly introduce the function of the extern "C" method. For specific explanations, please refer to this blog, which is very detailed and recommended for reading. For example, in C language, there is a function

int add(int a,int b);

If you use the gcc compiler, the name generated by the compilation is add, but if you use the g++ compiler, the name generated by the compilation may be something like ABaddCD, which includes the function name, number of input parameters, type, and return value. So, if C++ also defines an overloaded

float add(float a,float b);

The name generated by the compilation may be something like EFaddGH, which also contains information such as function name, input parameters, return value, etc., so C++ can be overloaded. Imagine if you use the gcc compiler, then they are all called add, and you can't tell which function it is, so you can't overload it. Then, the role of extern "C" is to tell the g++ compiler to compile int add(int a, int b) into add instead of ABaddCD, because add can be recognized by C language, while ABaddCD cannot be recognized by C language, and C language will think that add is an 'undefined symbol'. So, from here we can also see that extern "C" can only be used for C++ code. In addition, for overloaded C++ functions, you need to write two different functions to call them separately to ensure that the names are not repeated.

Python uses extern "C" to call C++ dynamic library

After knowing the nature of extern "C", we will encapsulate it according to this method. I directly took the source code of the C++ dynamic library, encapsulated a layer of C interface on top of the source code, and then generated the dynamic library. Assume that the add function is encapsulated as addc, the C++ dynamic library is called A, and the dynamic library generated after encapsulating a layer of C interface is called B. If you write a test code named test.c, use pure C code to check dynamic library B, and call the addc function, the result is feasible and successful. However, when using Python to check dynamic library B and calling the addc function, the following error is reported:

AttributeError: B.so: undefined symbol: add

That is to say, the add function is still not recognized. use

nm B.so | grep add

Be able to get

addc
ABaddCD

As a result, the first addc can definitely be recognized by Python, but the second ABaddCD is the name generated by g++ compilation and cannot be called by Python. I am just giving an example from my own experience. The source code of my own C++ dynamic library may be relatively complex and cannot be successfully called by Python. There are many examples online that can be successfully called. So readers can experiment on their own. If they can call it successfully, that would be the best. Because the way to use boost.python that I will introduce next is rather tortuous.

Python uses boost.python to call c++ dynamic library

Solve other third-party libraries that c++ dynamic libraries depend on

Since my dynamic library depends on other third-party library files, such as openssl, uuid, libevent, and pthread, no matter which method is used to call the C++ dynamic library, Python is required to load these dynamic libraries. The specific Python code is as follows:

from ctypes import *
ctypes.CDLL("libssl.so", mode=ctypes.RTLD_GLOBAL) 
ctypes.CDLL("libcrypto.so", mode=ctypes.RTLD_GLOBAL) 
ctypes.CDLL("libuuid.so", mode=ctypes.RTLD_GLOBAL) 
ctypes.CDLL("/usr/lib64/libevent.so", mode=ctypes.RTLD_GLOBAL) 
#ctypes.CDLL("/usr/lib64/libpthread.so.0", mode=ctypes.RTLD_GLOBAL)

Some can be loaded by default, such as libpthread.so, which we do not need to load. Others need to be loaded manually, such as libssl.so and libuuid.so, which are all in the /usr/lib64/ directory and do not need to be added to the path. However, the libevent library is also in the /usr/lib64 directory and is also in the /usr/lib/ directory, so the path must be added. So, if the compilation fails, use whereis libevent.so to check which directory it is in, and then add the absolute path. Sometimes adding the absolute path still does not work, for example, libpthread.so still reports an error after adding the absolute path

'OSError: /usr/lib64/libpthread.so: invalid ELF header'

This means that the version number is incorrect. Find the version number of the libpthread.so link and add the .0 version number, then no error will be reported.

C++ code configuration boost environment

On the centos6.6 machine where the c++ dynamic library is located, I refer to: Ubuntu python calls C/C++ method dynamic link library configuration and test boost. Reference: Using Boost.Python to implement Python C/C++ mixed programming to implement Python-defined C++ function overloading. When configuring the environment, the commands I used were: yum install boost*, yum install python-devel. I implemented boost by referring to these two articles, and basically all of them passed. The problems I encountered were also mentioned in these two articles. I also encountered other problems and found solutions on Stack Overflow. I will post the results directly below:

Create a new test.cpp, in which we need to define functions available to python.

In the test.cpp code, include the following code:

// Need to include boost header file #include <boost/python.hpp> 
#include <boost/python/module.hpp> 
#include <boost/python/def.hpp>

//Implementation of overloaded functions. In my C++ code, LOGIN function, Synchronize_Request function, and Notify function all have three overloaded functions. Below I only use one LOGIN function, one Synchronize_Request function, and two Notify functions. For example, fun3 and fun4 below are two different notify functions.

//Only overloaded functions need to define fun1, fun2, fun3, fun4 like this. If there is no overloaded function, you can directly write the name int (*fun1)(const int server_type, const int request_no, std::string& login_result) = &LOGIN;

int (*fun2)(const int server_type, const int request_no,std::string& recv_answer) = &Synchronize_Request; 
int (*fun3)(const int server_type, unsigned int timeout_ms, unsigned int sesscare ) = &Notify;

int (*fun4)(void) = &Notify;

// add function overloading example int (*fun5)(int a,int b) = &add;

 
BOOST_PYTHON_MODULE( libB ) //python module, the name of libB should be consistent with the name of .so { 
 using namespace boost::python;

 //The Initialize function is not overloaded, so you can use it directly without defining fun1 as above 
 def("Initialize",Initialize);
 //Uninitialize function is not overloaded, just use it directly def("Uninitialize",Uninitialize); 
 def("LOGIN",fun1); 
 def("Synchronize_Request",fun2); 
 def("Notify",fun3); 
 def("Notify2",fun4); 
 def("add",fun5); 
 // Python can call the function defined in def above}

The commands used by the Makefile are:

%.o : %.cpp
 g++ -g -lssl -fPIC -levent -lcrypto -luuid -lpthread -lrt -lboost\_filesystem -lboost\_system -lboost_python -lpython -I/usr/include/python2.7 -o $@ -c $<

The command to generate B.so is:

g++ -shared -Wl,-soname,libB.so -o libB.so *.o -lpython -lboost_python

The dynamic library needs to be introduced in the python script

import libB

print libB.add(10,20)

By writing and compiling according to the above commands, you can avoid the pitfalls I encountered. Pay attention to the position of -lpython, don't put it in front. If the overloaded definition is not implemented and def("LOGIN",LOGIN); is used directly, the following error will be reported: error: no matching function for call to 'def(const char [15], <unresolved overloaded function type>)' def("LOGIN",LOGIN); In summary, this is the result of my whole day's research. If there are any errors or omissions, please point them out. Thank you.

Supplement: When using boost.python to call the C++ dynamic library, I cannot handle reference types. For example, string& recv_answer is used to receive the return result and is recognized as string{lvalue}, but my python passes in string type and cannot match. So I manually changed the string type reference of string& recv_answer into char * recv_answer_c format, that is, changed it to C language style, and then passed the recv_answer_c parameter in the following way to receive the result.

#Use bytes to pre-allocate space for variables to ensure that there will be no segmentation errors temp = bytearray(1000)
recv_answer_c = bytes(temp)

Summarize:

The above is the full content of this article. I hope that the content of this article will have certain reference learning value for your study or work. If you have any questions, you can leave a message to communicate. Thank you for your support for 123WORDPRESS.COM.