Node and Python two-way communication implementation code

Third-party data suppliers encapsulate data and Python together, and data queries can only be implemented by calling Python methods. If Python method calls can be implemented through simple Node encapsulation, it can be quickly launched and development costs can be saved.

The simplest and crudest communication method is for Nodejs to call a Python script and then obtain the output of the child process. However, since the process of starting Python and loading data packets each time is relatively long, this process is optimized.

Process Communication

index.py

# Encapsulated Python package, huge size from mb import MB
# Query from the data packet mbe.get('1.0.1.0')

index.js

const { spawn } = require('child_process');
const ls = spawn('python3', ['index.py']);

ls.stdout.on('data', (data) => {
  console.log(`stdout: ${data}`);
});

ls.stderr.on('data', (data) => {
  console.error(`stderr: ${data}`);
});

ls.on('close', (code) => {
  console.log(`child process exited with code $[code]`);
});

Use child_process.spawn to spawn a Python subprocess and listen to stdout output. The above method is also an example in the official documentation. Currently, there are two problems with this example:

• Nodejs does not send data to Python
  
• After the Nodejs call is completed, the Python subprocess will exit; the next query needs to call the Python command again to load the file and query the data; it is not possible to load the memory once and use it multiple times.

Bidirectional communication between processes

The premise for ensuring that data can be loaded once and used multiple times is that the Python process cannot be exited after it is started. The reason why the Python process exits is because there is nothing to do, so common methods include loops, sleep, and listening to ports. These methods can be translated into synchronous blocking tasks and synchronous non-blocking tasks. The least expensive of them is synchronous non-blocking tasks. Then you can think of Linux's select and epoll. I did a simple search for Python's epoll and it seems that there is a native package.

index.py - listens to stdin via epoll

import sys
import fcntl
import select
from mb import MB
import json

mbe = MB('./data')

# epoll model fd = sys.stdin.fileno()
epoll = select.epoll()
epoll.register(fd, select.EPOLLIN)

try:
    while True:
        events = epoll.poll(10) # Synchronous non-blocking data = ''
        for fileno, event in events:
            data += sys.stdin.readline() # Get data through standard input if data == '' or data == '\n':
                continue
            items = xxx # Number processing for item in items:
                result = mbe.get(item)
                sys.stdout.write(json.dumps(result, ensure_ascii=False) +'\n') # Write to standard outputsys.stdout.flush() # Buffer refreshfinally:
    epoll.unregister(fd)
    epoll.close()

index.js - Send data via stdin

const child_process = require('child_process');
const child = child_process.spawn('python3', ['./base.py']);

let callbacks = [], 
    chunks=Buffer.alloc(0), 
    chunkArr = [], 
    data = '', 
    onwork = false; // buffer cannot be expanded dynamically child.stdout.on('data', (chunk) => {
    chunkArr.push(chunk)
    if (onwork) return;
    onwork = true;
    while(chunkArr.length) {
        chunks = Buffer.concat([chunks, chunkArr.pop()]);
        const length = chunks.length;
        let trunkAt = -1;
        for(const [k, d] of chunks.entries()) {
            if (d == '0x0a') { // 0a ends data += chunks.slice(trunkAt+1, trunkAt=k);
                const cb = callbacks.shift();
                cb(null, data === 'null' ? null : data )
                data = '';
            }
        }
        if (trunkAt < length) {
            chunks = chunks.slice(trunkAt+1)
        }
    }
    onwork = false;
})

setInterval(() => {
    if (callbacks.length) child.stdin.write(`\n`); // There is no flush method for standard input and output on the Nodejs side, so you can only use a hack. Python cannot get the latest data in time after writing.}, 500)

exports.getMsg = function getMsg(ip, cb) {
    callbacks.push(cb)
    child.stdin.write(`${ip}\n`); // Write data to the standard input of the child process}

Python and Nodejs communicate through stdio; Python listens to stdin through epoll to achieve memory residency and long-term operation.

Problems

• Nodejs uses standard output as the execution result, so the Python side can only write the execution result to standard output without any additional printing information.
• The standard input on the Nodejs side does not have a flush method, so the event triggering on the Python side is not timely enough. Currently, a hack is implemented by sending empty information periodically on the Nodejs side.
• Buffer cannot be expanded dynamically, is not as useful as C language pointers, and is ugly when parsing stdout

Summarize

Although two-way communication between Nodejs and Python can be achieved, due to the above-mentioned problems, this method is not recommended here. HTTP or Socket is much better.

This is the end of this article about the implementation code of two-way communication between Nodejs and Python. For more relevant content about two-way communication between Nodejs and Python, please search for previous articles on 123WORDPRESS.COM or continue to browse the following related articles. I hope you will support 123WORDPRESS.COM in the future!