Building OpenSCADA and firmware for ARM-controllers of ICP DAS company (LP-5141)
| Name: LP-5xxx Start: august 2011 Finish: september 2011 Version: 1.0.0 Performers: Roman Savochenko Description: The project is dedicated to building the OpenSCADA and firmware for ARM controllers of the LP-5xxx and LP-8x{3|4}x series of the ICP DAS company. Materials:  ftp://ftp.oscada.org/OpenSCADA/PLC/LP5xx1 |  |
Introduction
The ICP_DAS company produces the controllers on ARM architecture processors for quite a long time with OS Linux as preinstalled environment, which makes interesting the creation or OpenSCADA build or the new firmware with OpenSCADA.
Until the last moment there were a number of reasons that prevent the creation of an OpenSCADA build for ARM-controllers of the ICP DAS:
- absence of the ARM-controller;
- unavailability of OpenSCADA to work on ARM-architecture;
- extremely outdated Linux environment on the ARM-controllers of "ICP DAS".
At the moment, all of these obstacles have been eliminated. So, thanks to Peter Litkovets we got the LP-5141 controller, and by this time OpenSCADA was largely already adapted to work on ARM-architecture, through processing and stabilization on the handheld computer of Nokia company - N800. Later during the same work with the LP-5141 controller it has been eliminated the last obstacle by updating the original software environment while maintaining the original functionality.
In general, the goal of this project is to develop the mechanisms and the creation of OpenSCADA building for the ARM family of the controllers of "ICP DAS" company, as well as the creation of a firmware for the "ICP DAS" controllers, which will fall into the hands of OpenSCADA developers.
1. PLC LP-5141
PLC (Fig. 1) is designed in the monoblock form-factor that does not include built-in computer-process interface, except for the possibility of installing a special expansion board IO, and the connection of the external one is made through the built in RS-485, for example, in the form of modules I-7000 of the the "ICP DAS" company.
Fig. 1. PLC of the LP-5xxx series.
CPU of the controller has the following specifications:
| CPU | PXA270 or compatible (32-bit and 520MHz) |
| SDRAM | 128 MB |
| Flash | 64 MB |
| EEPROM | 16 KB Data storage: 40 years; 1,000,000 delete/write cycles. |
| Extended Flash memory | microSD socket with a 2GB microSD card (can support 16 GB microSDHC card) |
| 64-bit serial number of the equipment | Present |
| Dual Watchdog Timer | Present |
| VGA | 800 × 600 |
| Ethernet ports | RJ-45 x 2, 10/100 Base-TX Ethernet controller (Auto-negotiation, auto MDI/MDI-X, LED indicator) |
| USB 1.1 (хост) | 1 |
| COM1 (/dev/ttySA0) | RS-232 (RxD, TxD and GND); not isolated |
| COM2 (/dev/ttySA1) | RS-485 (D2+,D2-); 2500 VDC; isolated |
| COM3 (/dev/ttySA3) | RS-232 (RxD, TxD and GND); not isolated |
| Operating temperature | -25 ~ +75 °C |
2. Software environment based on OpenSCADA.
Software environment, based on the Linux operating system for ARM controllers of ICP DAS usually includes: the Linux kernel 2.6.19, GLibC 2.2.5 and GCC 2.95. To build a third-party software the ICP DAS provides an SDK with the cross-compiler, set of libraries and headers for them (Linux toolchain). As you can see, the version of GCC and are very old, namely it is the 2001 year. OpenSCADA building in this environment is actually impossible. If the using of the GLibC of 2.2.5 version is still possible, the GCC of version 2.95 has the C++ compiler, which crashes during the building of the OpenSCADA code, and the standard C++ library or STL is extremely limited and it requires the significant and senseless adaptation. For this reason, it is necessary to update the original software environment to version for at least 3 of GCC compiler and C++ library.
Because internal flash memory has a relatively small capacity (64 MB), and a full rebuild of the original software environment of the controller - a very laborious process, it was decided to ensure the compatibility of the basic libraries of old and new environments. Namely, the ability to use libraries like fontconfig, i8k from the source environment and the work of the source environment programs with the new libraries.
In the process of selecting a new software environment for the PXA-270 processor of the "ICP DAS" controller several options have been tried because some of them had one or other disadvantages:
- crosstool-0.43 — interesting but outdated project (latest version is 2006 year) to build a Linux cross environment for the necessary equipment. It was postponed after several unsuccessful attempts to build a new environment to work with "Software FP" and "VFP". As it turned out subsequently for the original environment object files the information about the presence there "Software FP" and "VFP" is incorrectly displayed, when in fact the operations with real numbers there are made through "FPA". Subsequently the problem solve, described in section "Remarks", by aid the tool have been built ToolChain grounded on profile "arm-xscale.dat" and "gcc-4.0.2-glibc-2.3.6-tls.dat", which at last have been used on base of new firmware.
- PXA-Linux Project — a project to create an independent set of Linux tools for building for the PXA processor. It has the binary building
02-25-2005/bin/arm-linux-toolchain-bin-12-15-04-driscoll.tar.gz. It turned out that this building was built with the "Software FP" and "VFP", and also has a problem in the form of disabling of set of expansions for the gcc compiler, for example: __attribute__((packed)).
- Voipac — binary build of the cross compilation tools (arm-linux-gcc-3.4.1.tar.gz) from the firm "Voipac" for their own equipment on the PXA-270 processor. This ToolChain built by help early seen crosstool.
Building OpenSCADA with use cross-compiler make next:
# Unpack instruments archive
$ cd /opt
$ tar --lzma -xvf lincon.tlz
# Initialize build environment
$ . /opt/lincon/linpac_new.sh
# Go to sources tree of OpenSCADA, configuration and build
$ cd /lincon/OpenSCADA
$ ./configure --host=arm-xscale-linux-gnu CXXFLAGS="-O2 -D_REENTRANT" CFLAGS="-O2 -D_REENTRANT" --disable-QTStarter --disable-QTCfg --disable-Vision --disable-MySQL --disable-FireBird --disable-PostgreSQL --disable-SoundCard --enable-ICP_DAS
$ make
The resulting environment of the controller with OpenSCADA was formed by replacing the original environment core libraries on the libraries from new tools, building of the OpenSCADA and placing the OpenSCADA files to source environment tree.
OpenSCADA was built with disabling of the number of libraries and modules. So, it were build and tested the following modules and functions of OpenSCADA:
- OpenSCADA core — built with the LibGD2 library (the work is checked). In the process of checking and adaptation the functions of storage the OpenSCADA configuration in the configuration file are expanded.
- DB.SQLite — SQLite DB (the work is checked). The library and module of OpenSCADA to work with SQLite database are built. During the check process the error of GLibC 2.3.2, related to the lack of check on inter-locking of "rw"-resources is found and eliminated.
- DB.DBF — DBF DB (the work is checked). Module to work with DBF-files of version 3.
- Transport.Sockets — The module the the TCP, UDP and Unix sockets' transport (the work is checked). In the process of TCP-sockets using it was found and corrected the structure's of check the socket's status not initialization.
- Transport.SSL — Transport module of the secure socket layer: SSL, TLS (the work is checked).
- Transport.Serial — Transport module of serial interfaces (the work is checked). It was found that for some reason COM3 is located on the device /dev/ttySA3.
- Protocol.SelfSystem — The module of the own OpenSCADA protocol (the work is checked).
- Protocol.HTTP — The module of the HTTP protocol implementation (the work is checked).
- Protocol.ModBus — The module of the ModBus protocol implementation (the work is checked). The configuration of the station as a data server through the ModBus protocol.
- Protocol.OPC_UA — The module of the OPC UA protocol implementation (the work is checked). The configuration of the station as a data server through the OPC UA protocol . There was a problem of difference of real number representation on x86 and ARM FPA!
- Protocol.UserProtocol — The module of free implementation by the user of simple protocols using the programming language of OpenSCADA (the work is checked).
- DAQ.System — The module of the operating system data (the work is checked). It is built without the LibSensors library using.
- DAQ.JavaLikeCalc — The module of implementation of user Java-like language and calculations based on it (the work is checked).
- DAQ.BlockCalc — The module of implementation of block computing (the work is checked).
- DAQ.LogicLev — The module of implementation of the logic level data sources (the work is checked).
- DAQ.ModBus — The module of working with data sources via the ModBus protocol (the work is checked).
- DAQ.DCON — The module of working with data sources via the DCON protocol (the work is checked). For example, with the devices ICP_DAS I-7000. It was found and fixed the crash when receiving a negative response size. The result of the answer is now always returned >=0.
- DAQ.ICP_DAS — The module of the ICP_DAS data sources (there is no equipment to check). Serial series: I-87000 and I-7000; Parallel (quick): series I-8000. The module is built with a library libi8k.a for ARM-architecture.
- DAQ.DAQGate — The module of implementation of the locking of data sources from other OpenSCADA stations (the work is checked).
- DAQ.OPC_UA — The module of working with data sources via the OPC_UA protocol (the work is checked). There was a problem of difference of real number representation on x86 and ARM FPA - the conversion when transmitting is added!
- DAQ.Siemens — The module of the working with the Siemens controllers via Industrial Ethernet (ISO_TCP) (the work is checked).
- DAQ.SNMP — The data acquisition module of the network equipment via SNMP (the work is checked).
- Archive.DBArch — The module of the database archiving (the work is checked).
- Archive.FSArch — The module of the file system archiving (the work is checked). There was a problem during the verifying of the archive, giving the following message at startup: "Error archive file structure: <ARCHIVES/VAL/1s/CPULoad_load 2011-09-12 17:47:17.val>. Margin = -8 byte. Will try fix it!". As a result, it was found that the algorithm of fast bits' counting is not aligned to 4 bytes when reading the memory buffer - it is fixed.
- UI.WebCfg — The Web configurator module of OpenSCADA (the work is checked).
- UI.WebCfgD — The dynamic Web configurator module of OpenSCADA (the work is checked).
- UI.VCAEngine — The visualization engine module (the work is checked).
- UI.WebVision — The module of the user interfaces visualizer in the UI.VCAEngine by means of the Web-interface (the work is checked).
- UI.WebUser — The module of free user's visualization by means of the Web-interface (the work is checked).
- Special.FLibComplex1 — The library of compatibility functions with Complex1 ("DIYA" LTD) (the work is checked). It is used for the block programming.
- Special.FLibMath —The library of standard mathematical functions (the work is checked).
- Special.FLibSYS — The library of system expansion functions of OpenSCADA (the work is checked).
For the purpose of convenient distribution and use of the resulting OpenSCADA build for ARM-controllers of "ICP DAS" the parsing of the firmwares format for these controllers and the formation of a new one, including OpenSCADA were completed. The firmware of the "ICP DAS" ARM-controllers is a binary file with a name like "lp5x4x_1.1.bin". Structurally, this file contains the kernel and root file system - JFFS2. You can get their exact location by using the commands in the original environment of the controller, for example, for LP-5141:
$ cat /proc/mtd
dev: size erasesize name
mtd0: 00040000 00020000 "Bootloader"
mtd1: 00040000 00020000 "Bootloader Param"
mtd2: 00280000 00080000 "Kernel"
mtd3: 03c80000 00080000 "JFFS2 Filesystem"
From the obtained result of the command you can conclude that the connection point of the image kernel and filesystem is placed at 0x280000 offset of the firmware file, and the size of the erase flash block is 0x80000 (512kB). With this information we can extract the firmware and make changes to the image of its file system:
# Extracting the source images of the kernel and root file system
$ dd if=lp5x4x_1.1.bin of=kernel_orig bs=1K count=2560
$ dd if=lp5x4x_1.1.bin of=jffs2_orig bs=1K skip=2560
# Connecting the the source image of the file system
# Create a block device emulation in memory.
# May cause an error in the case of use continuous memory area by other programs and its size small.
# To resolve this, add the kernel command line when booting "vmalloc=512MB".
$ modprobe mtdram total_size=65536 erase_size=512
$ modprobe mtdblock
$ dd if=root_orig.img of=/dev/mtdblock0
$ mount -t jffs2 /dev/mtdblock0 /mnt/tmp
# Copy the changes and OpenSCADA files to the root file system of the controller
# Create a new file system with changes.
$ mkfs.jffs2 --root=/mnt/tmp --pad=$((61952*1024)) --eraseblock=512 --output=root.img
# Build the new firmware image
$ cat kernel_orig.img root.img > lp5x4x_oscada_1.1.bin
Thus we get the new firmware with OpenSCADA in the file lp5x4x_oscada_1.1.bin, which can be downloaded to the controller in the standard way, described in the "ICP DAS" documentation.
3. Remarks
As it turned out that the operations with real numbers are performed by the FPA co-processor instructions (Float Point Acceleration), which is not present in this processor, and calls of the commands themselves are made through an imitation of an exception in the Linux kernel, then the performance of mathematical calculations is extremely low, even compared to the direct "Software FP", "VFP". For example, the calculation of a single operation sin(pi) is about ~200 ms, compared to ~20 ms for N800 with VFP and ~2 ms at x86, detailed in table by the link (RU). So, this controller shouldn't be considered as the platform for at least any serious calculations! Moreover, the real numbers when working with FPA are stored in a different way - namely, the special big-endian, which requires a transformation in the case of the binary external exchange to the typical little-endian representation, for example for DAQ.OPC_UA.
In the initial configuration the COM1 port (/dev/ttySA0) is used as a console, which is initialized by the kernel command "console=/dev/ttySA0". However, after adjustment this is not necessary and is often desirable to release another one COM-port for other purposes. It is possible to release COM1 from work for a console by reassigning the other device for this role, for example by: $busybox setconsole /dev/tty1. However, this method does not fully release the COM-port and requests are lost, reading by the parallel connection. Probably need to edit the kernel boot options line in the bootloader U-Boot, even enter into it possible only in the position of RS = 2, however for which the initialization parameters are separated from the main mode of RS = 0. It is necessary to achieve the entrance to the U-Boot from the primary mode of RS = 0!
Translation is going now
In the process of using the created software environment with OpenSCADA it was found a strange problem: crashing the OpenSCADA during access to an instance of the resource string from different threads. The problem is reproduced when polling the controller via ModBus/RTU with 100 ms frequency, the polling an absent module by DCON on the same RS485 interface as the ModBus/RTU. At the same time the periodic (1 sec) record of two registers to the polled via ModBus/RTU controller is made. The crash does not occur immediately, but within 10 minutes - 5 hours. Analysis of the problem:
(*) It was found that the crashes begin in case of a achievement by TMdContr::modBusReq() function of DAQ.ModBus module, namely, concurrent access to controller's object property "mPrt" in the context of the "XMLNode req (mPrt);" constructor from different threads of reading and writing.
(*) To clarify the nature of the problem it has been configured the generation of death memory dump, as well as assembled the gdb cross-debugger. During the study of death dump it was found that there is a destruction of the stack, and there is no rational reasons for that in OpenSCADA, so this is a problem of the system environment for specific conditions.
(*) When you access the "mAddr" property, in the first line of the TMdContr::modBusReq() function and in the problem context of "XMLNode req (mPrt);" (with an incomplete record), the crash is not seen.
(*) Проверка вызова второй строки "XMLNode req(mPrt);", при записи и в целом, с постоянной "RTU". При замене на "RTU" в целом падение замечается уже в другом месте. При замене только неполной записи падение не замечается. Вторым и последним местом падения оказалась следующая строка req.setAttr("id",id()). Общее между ними то, что осуществляется доступ к элементу конфигурации строкового типа, использующий объект ресурсного типа "ResString". Вывод, текущее программное окружение некорректно работает с объектом "ResString" при доступе на чтение (немодифицирующий) из двух разных потоков.
(*) Способ хранения объекта "ResString" в элементе конфигурационного поля "TCfg" значения не имеет, а именно — падение наблюдается при прямом хранении "ResString" в "TCfg", а так-же при прямом обращении к объекту "string" внутри "ResString", в случае с "id()".
(*) Проверка при отключении охвата ресурсом — проблемы не решает.
(*) Сборка нового окружения с помощью "PTXDist", и запуск его через "chroot" — при запуске EABI окружения, для TionPro270 из под chroot, происходит ошибка "Illegal instruction!". Не запускается EABI ядро от TionPro270 c похожей ошибкой. Собранный ToolChain из профиля "arm-xscale_hardfloat-linux-gnu_gcc-4.0.4_glibc-2.3.6_binutils-2.17_kernel-2.6.18.ptxconfig" в "OSELAS.Toolchain-2011.11.0" падает с ошибкой сегментации, при запуска OpenSCADA собранной под исходное окружение.
(*) Сборка OpenSCADA с включением всех нужных модулей в библиотеку ядра OpenSCADA проблему так-же не решает.
(*) Осуществлена проверка той же конфигурации на другом PXA270-устройстве (TionPro270) для окончательного выяснения источника проблемы (аппаратного или программного окружения) — конфигурация проработала беспроблемно двое суток.
(*) Заменить в объекте TCfg "ResString" на обычный "string" с выносом ресурса доступа в объект контейнера TConfig, за одно и уменьшив потребление памяти посредством обобщения ресурса доступа к объекту строки — замена осуществлена, проблема осталась.
(*) Вычитать сборку с заменой "ResString" на обычный "string" в TCfg и проверить место текущего падения — место падения тоже.
(*) Взять "crosstool" и собрать несколько более свежий toolchain — собран toolchain "gcc-4.0.2-glibc-2.3.6-arm-xscale-linux-gnu", проблема воспроизводится, скорее всего проблема в работе с памятью текущего ядра. В процессе изучения найдены прецеденты с подобными проблемами в glibc-2.2 — 2.7, которые содержат реализацию malloc небезопасную для потоков (nothread-safe).
(*) Замена аллокатора памяти — не удалось найти рабочей и при этом прозрачной реализации стороннего аллокатора, ptmalloc3 просто падает.
(*) Сборка ToolChain и OpenSCADA со всеми возможными параметрами включения thread-safe — во всех вариантах проблема присутствует, а именно на ToolChain с параметрами GLibC: "--with-tls --with-__thread" и OpenSCADA собрана с параметрами: "-pthread -D_REENTERANT".
(*) Перехват обращений аллокации памяти и оборачивание глобальным ресурсом, функций "С" malloc, free и " new, delete — падения наблюдаются.
(+) Проверка гипотезы проблемности реализации (атомарности) COW (Copy on Write) алгоритма в объекте "string" — копирование из объекта "string" хранилища выполнено посредством val.c_str(), что исключает выполнение COW: подтверждено, что данная проблема связана с алгоритмом COW, а именно с атомарностью доступа к счётчику строк "_Atomic_word _M_refcount;", вероятно из-за устаревшей системы потоков linuxthread.
4. Conclusion
The result was the building of OpenSCADA for Linux controllers of ARM-architecture of "ICP DAS" company. The building can be loaded directly on any Linux-ARM controller, unpacking it at the root of the original operating system. In this case, however, the duplicates of the old core libraries (/lib/*) will remain, which can be removed after a successful restart.
For family of controllers LP-5x4x the firmware with the building OpenSCADA was created, which can be downloaded in the standard for these controllers way, which is described in the original "ICP DAS" documentation. Firmware for LP-5x4x can possible fit for the LP-5x3x, but it is not checked!
Due to outdated and non-optimal initial building of the Linux environment from "ICP_DAS", which imposes the restriction on freedom of building the optimal environment, after the completion of the project it is planned to apply for support to "ICP_DAS" with the request to correct this situation.
Links
- Archive of the OpenSCADA building: ftp://ftp.oscada.org/OpenSCADA/PLC/LP5xx1/root_oscada.tgz
- Firmware with OpenSCADA for LP5x41: ftp://ftp.oscada.org/OpenSCADA/PLC/LP5xx1/lp5x4x_oscada_1.1.bin.gz