The rtorrent v0.8.1 from the Optware on a Buffalo Linkstation Live (LS-CHLFFE) v2 NAS doesn’t support TLSv1.2. We need to upgrade rtorrent to a recent release in order to support HTTPS trackers requiring TLSv1.2. This post is about the process of building rtorrent v0.9.6 from the source code.

Introduction

This post is about building rtorrent 0.9.6 on a Buffalo Linkstation Live (LS-CHLFFE) v2 NAS. It’s assumed that Optware has been set up. This means that necessary compilation tools can be installed by using ipkg command.

We are going to build the following software components:

• automake 1.16
• libz v1.2.1
• OpenSSL v1.1.0h
• curl v7.59.0
• libtorrent v0.13.6
• rtorrent v0.9.6

Optionally, rtorrent supports XML-RPC, which requires xmlrpc-c. To build xmlrpc-c, a recent version of Boost is required. Since I don’t use XML-RPC functionalities, in this post, rtorrent will be configured without XML-RPC.

Building steps

All the tools will be configured with prefix=/opt/local. This means that compiled built binaries and support files will be installed in subdirectories in /opt/local, for instance, executables will be installed in /opt/local/bin, and libraries in /opt/local/lib. We need to add /opt/local/bin and /opt/local/lib to the font of $PATH and $LD_LIBRARY_PATH respectively.

automake

Change to the directory where automake-1.16.tar.gz is saved.

tar zxvf automake-1.16.tar.gz
cd automake-1.16
mkdir mybuild
cd mybuild
../configure --prefix=/opt/local
make

# as root
make install

libz

Change to the directory where zlib-1.2.11.tar.gz is saved.

tar zxvf zlib-1.2.11.tar.gz
cd zlib-1.2.11
mkdir mybuild
cd mybuild
../configure  --prefix=/opt/local
make

# as root
make install

OpenSSL

Change to the directory where openssl-1.1.0g.tar.gz is saved.

tar zxvf openssl-1.1.0h.tar.gz
cd openssl-1.1.0h
mkdir mybuild
cd mybuild
../config -Wl,--enable-new-dtags --prefix=/opt/local/ssl --openssldir=/opt/local/ssl enable-egd
make

# as root
make install

NB: The prefix is set to “/opt/local/ssl”, therefore, the path should be add to $LD_LIBRARY_PATH as well. The configuration needs to include “enable-egd”, otherwise, curl would fail to compile.

curl

Change to the directory where curl-7.59.0.tar.gz is saved.

tar zxvf curl-7.59.0.tar.gz
cd curl-7.59.0
mkdir mybuild
cd mybuild
PKG_CONFIG_PATH=/opt/local/ssl/lib/pkgconfig ../configure --prefix=/opt/local --with-ssl=/opt/local/ssl --with-libssl-prefix=/opt/local/ssl
make

# as root
make install

libtorrent

Change to the directory where libtorrent-0.13.6.tar.gz is saved.

tar zxvf libtorrent-0.13.6.tar.gz
cd libtorrent-0.13.6
mkdir mybuild
cd mybuild
PKG_CONFIG_PATH="/opt/local/ssl/lib/pkgconfig:/opt/local/lib/pkgconfig" ../configure --prefix=/opt/local
make

# as root
make install

Here is where it gets tricky. The previous steps work normally. But the installation of libtorrent v0.13.6 would fail with errors like the following

undefined reference to '__sync_sub_and_fetch_4'

It has turned out to that the version of gcc used for the compilation doesn’t have atomic builtin functions for ARM. Basically, the default gcc, gcc 4.2.3, doesn’t have atomic builtins on ARM. More information can be found here and here.

In fact, libtorrent 0.13.6 can be configured not to use those atomic builtins by passing “-disable-instrumentation” to the configure script. That would allow libtorrent to be installed without any patches or tricks with gcc 4.2.3. However, rtorrent  also uses atomic builtins and it can be turned off in rtorrent. Rtorrent would fail to link even with the tricks mentioned in above two links.

So the solution would be to apply the tricks, that is, get the source code of those atomic builtins from gcc source tree, compile the code accordingly and link the compiled object code in both libtorrent and rtorrent. Change to a new working directory, for instance, ~/downloads/linux-atomic

wget -O linux-atomic.c "http://gcc.gnu.org/git/?p=gcc.git;a=blob_plain;f=libgcc/config/arm/linux-atomic.c;hb=master"
libtool --tag=CC --mode=compile gcc -g -O2 -MT linux-atomic.lo -MD -MP -MF linux-atomic.Tpo -c -o linux-atomic.lo linux-atomic.c
libtool --tag=CC --mode=link gcc -g -O2 -o liblinux-atomic.la linux-atomic.lo

Now change to the directory where the source code of libtorrent is. But before we start building libtorrent, as I have tested, we need to apply one more change to the source code of libtorrent, which is also related to the use of atomic builtins in libtorrent. Edit src/utils/instrumentation.h and src/utils/instrumentation.cc, change the all the instances of int64_t to int32_t, and PRIi64 to PRIi32. Then, let’s change to mybuild directory and build libtorrent again. Note, if you have already run configure before in mybuild directory, please run “make distclean” before running the following commands.

# Assume linux-atomic.c is in /home/taohe/downloads/linux-atomic
PKG_CONFIG_PATH="/opt/local/ssl/lib/pkgconfig:/opt/local/lib/pkgconfig" ../configure --prefix=/opt/local LDFLAGS="-L/home/taohe/downloads/linux-atomic" LIBS="-llinux-atomic"
make 

# as root 
make install

rtorrent

Compilation of rtorrent requires ncurses header files. In Optware, we need to install “ncurses-dev” and “ncursesw-dev”.

Change to the directory where rtorrent-0.9.6.tar.gz is saved.

# Assume linux-atomic.c is in /home/taohe/downloads/linux-atomic 
tar zxvf rtorrent-0.9.6.tar.gz
cd rtorrent-0.9.6
mkdir mybuild
cd mybuild
PKG_CONFIG_PATH="/opt/local/lib/pkgconfig:/opt/local/ssl/lib/pkgconfig" ../configure --prefix=/opt/local LDFLAGS="-L/home/taohe/downloads/linux-atomic" LIBS="-llinux-atomic"
make

# as root
make install

Done! For configuration of rtorrent, command list, and more information about rtorrent, please refer to rtorrent wiki page.

Download

The compilation takes a long time on the NAS. I’ve made an archive of /opt/local. The executables (rtorrent and curl) should be able to run on compatible systems. The archive can be downloaded here. See README file included for more information about the archive.

The post Build rtorrent v0.9.6 for Linux on ARM (armv5tel) with SSL and TLSv1.2 appeared first on Shajisoft.

Introduction

PNG is one of the commonly used image formats on the Internet. I used to save screenshots as PNG files. Unlike normal photos, screenshots of software applications usually have less colors. Using PNG, we can easily get images with good qualities and small file sizes. What’s more, it’s an open source format. For these reasons, I prefer PNG to other formats when creating screenshots.

To be honest, I hadn’t paid much attention to the actual sizes of PNG files generated by software tools until recently when I need to create some PNG files for my blog and web pages. I mainly use two software tools to create screenshots which are Gimp and Windows Paint. I assumed that these tools would generate PNG files of optimal sizes. Whereas, I noticed recently that Gimp and Windows Paint generate PNG of different sizes of the same screenshot. Although there are no options about PNG format in Windows Paint, it still creates smaller PNG files than Gimp with the highest compression level selected. Then I was curious about the difference. So I did some experiments. In this post, I am going to share something I have learned about PNG from the process.

PNG format

Here, PNG refers to Portable Network Graphics. It’s a raster graphics file format. It was created as a replacement to GIF which uses a patented compression algorithm LZW. Please refer to the Wikipedia for more details of PNG format.

Basically a PNG file starts with 8-byte header followed by chunks. Each chunk consists of four parts:

1. Length of the chunk – 4 bytes
2. Chunk type – 4 bytes
3. Chunk data – (Length of the chunk) bytes
4. CRC – 4 bytes

That is, if the first four bytes of a chunk is 8192 as an integer, it means that the data of the chunk has 8192 bytes, and the full size of the chunk is 4+4+8192+4 = 8204 bytes.

The second four bytes of a chunk indicates the type of the chunk, which is the case sensitive chunk type in ASCII format. A valid PNG file must have at least the following four critical chunk types:

1. IHDR: Must be the first chunk. It contains the meta information about the image like width, height etc
2. PLTE or sRGB: List of colors or the standard sRGB colors depending on the type of PNG
3. IDAT: Image data
4. IEND: Marks the end of the image

A PNG file can contain multiple IDAT chunks. To decode a PNG file, all IDAT chunks in a PNG file will be concatenated and the combined IDAT data is decompressed to give the image.

The following is a screenshot of TotalCommander Lister showing a PNG file in Hex format. We can see the first 8-byte header followed by IHDR and sRGB chunks:

Parse PNG file using Python

Firstly, let’s create a class to model chunks in a PNG file:

class Chunk(object):

    '''Definition of a chunk contained in a PNG file '''

    def __init__(self, buf):
        '''TODO: to be defined1. '''
        self.__length = 0
        self.__type = 0
        self.__data = []
        self.__crc = 0
        self.__size = 0     # The total size of the chunk in bytes

        if buf:
            self.load(buf)

    def load(self, buf):
        '''Load the chunk from the given buffer.

        @param buf List of bytes read from the PNG file
        @return: None

        '''
        length_buf = binascii.hexlify(buf[:4])
        #self.__length = int(''.join(length_buf), 16)
        self.__length = int(length_buf, 16)

        fmt = '4c'
        #self.__type = ''.join(struct.unpack(fmt, ''.join(buf[4:8])))
        self.__type = buf[4:8].decode()

        end_data = 8+self.__length
        self.__data = buf[8:end_data]
        
        buf = buf[end_data:]
        crc_bytes = binascii.hexlify(buf[:4])
        #self.__crc = int(''.join(crc_bytes), 16)
        self.__crc = int(crc_bytes, 16)

        self.__size = 8 + self.__length + 4;

    def totalSize(self):
        '''TODO: Returns the total size of the chunk in bytes.
        @return: TODO

        '''
        return self.__size

    def type(self):
        '''TODO: Docstring for type.
        @return: TODO

        '''
        return self.__type

    def data(self):
        '''TODO: Docstring for data.
        @return: TODO

        '''
        return self.__data

    def length(self):
        '''TODO: Docstring for length.
        @return: TODO

        '''
        return self.__length

To decode a PNG file, firstly open the file, read the content into a buffer, then decode the header and chunks:

class PNG(object):

    '''class for a PNG file. '''

    def __init__(self, file_path):
        '''TODO: to be defined1. '''
        self.__fileName = file_path
        self.__file = None
        self.__fobj = None
        self.size = [0, 0]
        self.__chunks = []  # List of chunks
        self.__width = 0
        self.__height = 0
        self.__bitDepth = 0
        self.__colorType = 0
        self.__cmpMtd = 0
        self.__fltMtd = 0
        self.__itlMtd = 0
        self.__totalIDAT = 0

        if file_path:
            self.readFile(file_path)
            self.__file = file_path
    
    def loadHeader(self, buf):
        '''TODO: Docstring for loadHeader.

        @param buf Byte buffer read from the import file
        @return: TODO

        '''
        pass

    def readFile(self, file_path):
        '''TODO: Docstring for readFile.

        @param file_path Full path to the png file
        @return: True if successful, False otherwise

        '''
        
        if file_path is None and self.__file is None:
          logger.info("File name is empty. Nothing to load!")
          return
          
        if file_path is not None:
          self.__file = file_path
        
        # Open the file for reading
        self.__fobj = open( self.__file, 'rb')

        fsize = os.path.getsize( file_path )
        # map the file into memory
        fno = self.__fobj.fileno()
        data = mmap.mmap( fno, fsize, access=mmap.ACCESS_READ )

        fmt = '8c'
        header = struct.unpack( fmt, data[:8] )

        # Get the header information
        #self.loadHeader() 
        #logger.debug("Header: %s" %  header )
        
        #working buffer   
        buf = data[8:]
        ihdr = Chunk(buf)
        self.parseIHDR(ihdr.data() )

        chunk_size = ihdr.totalSize()
        self.__chunks = [] 
        self.__chunks.append(ihdr)
        while len(buf) > chunk_size:
            buf = buf[chunk_size:]
            ck = Chunk(buf)
            chunk_size = ck.totalSize()
            self.__chunks.append(ck)
       
        self.parseIDAT()


        ### close the file
        data.close()
        self.__fobj.close() 
...

A valid PNG file contains at least one IDAT chunk. IDAT chunks contain compressed image data, which is a rectangular pixel array. Each pixel consists of color information. There are mainly three types of pixels in PNG files:

1. Grayscale pixel. Minimum pixel size: 1 bit
2. Color-indexed pixel. Minimum pixel size: 1 bit
3. Truecolor pixel: Minimum pixel size: 24-bit

Optionally, each pixel can have transparence information which is 8-bit. Therefore, a truecolor pixel with transparence consists of 4 bytes.

Image data is organized from the top to bottom. Each row of the image data is packed from the left to the right to form a scanline. Each scanline always starts with a single byte indicating the filter type followed by the packed pixel array of the image. All scanlines are then compressed by using DEFLATE algorithm which is the same as in zlib. We can use zlib module in Python to decompress data in IDAT chunk as shown below:

def testCompress(self):
        idat_data = b''
        self.__totalIDAT = 0
        for i in range(1, len(self.__chunks)):
            if self.__chunks[i].type() == 'IDAT':
                idat_data += self.__chunks[i].data() 
       img_data = zlib.decompress( idat_data ) 
       ...

Please refer to zlib RFC page for more details of the ZLIB compress data format specification. Basically zlib compressed data always has two bytes in the beginning to indicate compression information. For example, the first two bytes of zlib compressed data would appear to be “78 da” in hexadecimal. From these two bytes, we can guess some parameters used in DEFLATE algorithm. But can only get possible values for some parameters like compression level, because different compression level may result in the same header values in the compressed data. In order to figure out the actual compression parameters, we can use a try and error method to test the combination of parameters with commonly used values in Python.

There is already a Python module to parse PNG file. But because I want to test different data compression parameters, I wrote a Python script for the purpose. The complete Python script file can be downloaded from here. The downloaded Python script, sspng.py, works with Python 2.7 and 3.4. It accepts one argument as the path to a png file, dumps the image mata information, and tries to figure out the compression parameters used to create the PNG file. If failed, it prints the actual IDAT sizes if using the compression parameters used in the test.

Exampels

Let’s take a screenshot of TotalCommander on Windows 10 x64 as an example, which is shown below.

To take the screenshot, bring TotalCommander to the front on the desktop, press Alt-PrtScrn. It will copy the screenshot of TotalCommander to the clipboard. Then paste the image in applications which we want to use to edit and save the screenshot. Here, I am going to use Gimp v2.9 development version and the default Windows Paint. When saving the screenshot as a PNG in these two applications, there is no option to select in Paint regarding PNG file, whereas there are options including the compression level in Gimp to select. Take the default options in Gimp which means to use the highest compression level 9. The generated file sizes are shown below:

• Paint: 125,149 bytes
• Gimp: 162,688 bytes

Assume the PNG file of the above screenshot created by Gimp is saved as c:\users\wdong\Pictures\totalcmd-en-gimp.png, we can run sspng.py on it as shown below:

C:\Users\dongw\Pictures>c:\Python27\python.exe sspng.py totalcmd-en-gimp.png


Information about totalcmd-en-gimp.png
IHDR information:
        Width: 1010
        Height: 793
        Bit depth: 8
        Color type: 2
        Compression method: 0
        Filter method: 0
        Interlace method: 0

 Chunk pHYs: Data length: 9, total size: 21 bytes

 Chunk tIME: Data length: 7, total size: 19 bytes

 Chunk tEXt: Data length: 25, total size: 37 bytes

 Chunk IDAT: IDAT chunk size: 8192 bytes, total IDAT size: 162326 bytes

 Chunk IEND: Data length: 0, total size: 12 bytes
size of img_data = 2403583
zlib header in the PNG file = 78, da
With compression level 9, mem_level 8, strategy 1, zlib.compress gives 162326 bytes

From the above information, we know that the image height is 793 pixels. This means that there are 793 scanlines in IDAT chunk. The color type is 2 so there are 3 bytes (24 bits) per pixel. The first byte of each scanline contains filtering information. The uncompressed image data has (1+1010*3)*793=2403583 bytes. The compressed data is split into multiple IDAT chunks in the file. Each IDAT chunk contains 8192 bytes. The total compressed image data has 162326 bytes. It seems that Gimp uses the standard DEFLATE algorithm to compress the image data when creating PNG file, for we can get the same sized data by compressing the same image data using zlib module of Python.

Running the script on the screenshot created by Windows Paint shows the following information:

C:\Users\dongw\Pictures>c:\Python27\python.exe sspng.py totalcmd-cn-paint.png


Information about totalcmd-cn-paint.png
IHDR information:
        Width: 1010
        Height: 793
        Bit depth: 8
        Color type: 2
        Compression method: 0
        Filter method: 0
        Interlace method: 0

 Chunk sRGB: Data length: 1, total size: 13 bytes

 Chunk gAMA: Data length: 4, total size: 16 bytes

 Chunk pHYs: Data length: 9, total size: 21 bytes

 Chunk IDAT: IDAT chunk size: 65445 bytes, total IDAT size: 124688 bytes

 Chunk IEND: Data length: 0, total size: 12 bytes
size of img_data = 2403583
zlib header in the PNG file = 78, 5e
With compression level 0, mem_level 1, strategy 0, zlib.compress gives 2427294 bytes
With compression level 1, mem_level 1, strategy 0, zlib.compress gives 176182 bytes
With compression level 2, mem_level 1, strategy 0, zlib.compress gives 160285 bytes
With compression level 3, mem_level 1, strategy 0, zlib.compress gives 146284 bytes
With compression level 4, mem_level 1, strategy 0, zlib.compress gives 134058 bytes
With compression level 5, mem_level 1, strategy 0, zlib.compress gives 125760 bytes
With compression level 6, mem_level 1, strategy 0, zlib.compress gives 117158 bytes
With compression level 7, mem_level 1, strategy 0, zlib.compress gives 116340 bytes
With compression level 8, mem_level 1, strategy 0, zlib.compress gives 113287 bytes
With compression level 9, mem_level 1, strategy 0, zlib.compress gives 111966 bytes
With compression level 0, mem_level 2, strategy 0, zlib.compress gives 2415384 bytes
With compression level 1, mem_level 2, strategy 0, zlib.compress gives 170222 bytes
With compression level 2, mem_level 2, strategy 0, zlib.compress gives 155723 bytes
With compression level 3, mem_level 2, strategy 0, zlib.compress gives 143172 bytes
With compression level 4, mem_level 2, strategy 0, zlib.compress gives 129169 bytes
With compression level 5, mem_level 2, strategy 0, zlib.compress gives 122506 bytes
With compression level 6, mem_level 2, strategy 0, zlib.compress gives 114908 bytes
With compression level 7, mem_level 2, strategy 0, zlib.compress gives 114331 bytes
With compression level 8, mem_level 2, strategy 0, zlib.compress gives 111654 bytes
With compression level 9, mem_level 2, strategy 0, zlib.compress gives 110420 bytes
With compression level 0, mem_level 3, strategy 0, zlib.compress gives 2409474 bytes
With compression level 1, mem_level 3, strategy 0, zlib.compress gives 164668 bytes
With compression level 2, mem_level 3, strategy 0, zlib.compress gives 150928 bytes
With compression level 3, mem_level 3, strategy 0, zlib.compress gives 139918 bytes
With compression level 4, mem_level 3, strategy 0, zlib.compress gives 124791 bytes
With compression level 5, mem_level 3, strategy 0, zlib.compress gives 118426 bytes
With compression level 6, mem_level 3, strategy 0, zlib.compress gives 112056 bytes
With compression level 7, mem_level 3, strategy 0, zlib.compress gives 111606 bytes
With compression level 8, mem_level 3, strategy 0, zlib.compress gives 108879 bytes
With compression level 9, mem_level 3, strategy 0, zlib.compress gives 107891 bytes
With compression level 0, mem_level 4, strategy 0, zlib.compress gives 2406529 bytes
With compression level 1, mem_level 4, strategy 0, zlib.compress gives 161203 bytes
With compression level 2, mem_level 4, strategy 0, zlib.compress gives 148187 bytes
With compression level 3, mem_level 4, strategy 0, zlib.compress gives 137649 bytes
With compression level 4, mem_level 4, strategy 0, zlib.compress gives 121817 bytes
With compression level 5, mem_level 4, strategy 0, zlib.compress gives 115812 bytes
With compression level 6, mem_level 4, strategy 0, zlib.compress gives 110109 bytes
With compression level 7, mem_level 4, strategy 0, zlib.compress gives 109712 bytes
With compression level 8, mem_level 4, strategy 0, zlib.compress gives 107154 bytes
With compression level 9, mem_level 4, strategy 0, zlib.compress gives 106122 bytes
With compression level 0, mem_level 5, strategy 0, zlib.compress gives 2405059 bytes
With compression level 1, mem_level 5, strategy 0, zlib.compress gives 159511 bytes
With compression level 2, mem_level 5, strategy 0, zlib.compress gives 146528 bytes
With compression level 3, mem_level 5, strategy 0, zlib.compress gives 136218 bytes
With compression level 4, mem_level 5, strategy 0, zlib.compress gives 120177 bytes
With compression level 5, mem_level 5, strategy 0, zlib.compress gives 114230 bytes
With compression level 6, mem_level 5, strategy 0, zlib.compress gives 108545 bytes
With compression level 7, mem_level 5, strategy 0, zlib.compress gives 108230 bytes
With compression level 8, mem_level 5, strategy 0, zlib.compress gives 105560 bytes
With compression level 9, mem_level 5, strategy 0, zlib.compress gives 104583 bytes
With compression level 0, mem_level 6, strategy 0, zlib.compress gives 2404324 bytes
With compression level 1, mem_level 6, strategy 0, zlib.compress gives 158833 bytes
With compression level 2, mem_level 6, strategy 0, zlib.compress gives 146029 bytes
With compression level 3, mem_level 6, strategy 0, zlib.compress gives 135299 bytes
With compression level 4, mem_level 6, strategy 0, zlib.compress gives 120105 bytes
With compression level 5, mem_level 6, strategy 0, zlib.compress gives 114075 bytes
With compression level 6, mem_level 6, strategy 0, zlib.compress gives 108182 bytes
With compression level 7, mem_level 6, strategy 0, zlib.compress gives 107714 bytes
With compression level 8, mem_level 6, strategy 0, zlib.compress gives 104922 bytes
With compression level 9, mem_level 6, strategy 0, zlib.compress gives 103967 bytes
With compression level 0, mem_level 7, strategy 0, zlib.compress gives 2403959 bytes
With compression level 1, mem_level 7, strategy 0, zlib.compress gives 157836 bytes
With compression level 2, mem_level 7, strategy 0, zlib.compress gives 145142 bytes
With compression level 3, mem_level 7, strategy 0, zlib.compress gives 134822 bytes
With compression level 4, mem_level 7, strategy 0, zlib.compress gives 119127 bytes
With compression level 5, mem_level 7, strategy 0, zlib.compress gives 113359 bytes
With compression level 6, mem_level 7, strategy 0, zlib.compress gives 107632 bytes
With compression level 7, mem_level 7, strategy 0, zlib.compress gives 107275 bytes
With compression level 8, mem_level 7, strategy 0, zlib.compress gives 104603 bytes
With compression level 9, mem_level 7, strategy 0, zlib.compress gives 103742 bytes
With compression level 0, mem_level 8, strategy 0, zlib.compress gives 2403954 bytes
With compression level 1, mem_level 8, strategy 0, zlib.compress gives 157680 bytes
With compression level 2, mem_level 8, strategy 0, zlib.compress gives 145059 bytes
With compression level 3, mem_level 8, strategy 0, zlib.compress gives 134853 bytes
With compression level 4, mem_level 8, strategy 0, zlib.compress gives 118942 bytes
With compression level 5, mem_level 8, strategy 0, zlib.compress gives 113425 bytes
With compression level 6, mem_level 8, strategy 0, zlib.compress gives 107872 bytes
With compression level 7, mem_level 8, strategy 0, zlib.compress gives 107516 bytes
With compression level 8, mem_level 8, strategy 0, zlib.compress gives 104837 bytes
With compression level 9, mem_level 8, strategy 0, zlib.compress gives 103848 bytes
With compression level 0, mem_level 9, strategy 0, zlib.compress gives 2403954 bytes
With compression level 1, mem_level 9, strategy 0, zlib.compress gives 158120 bytes
With compression level 2, mem_level 9, strategy 0, zlib.compress gives 145273 bytes
With compression level 3, mem_level 9, strategy 0, zlib.compress gives 134947 bytes
With compression level 4, mem_level 9, strategy 0, zlib.compress gives 119473 bytes
With compression level 5, mem_level 9, strategy 0, zlib.compress gives 114064 bytes
With compression level 6, mem_level 9, strategy 0, zlib.compress gives 108411 bytes
With compression level 7, mem_level 9, strategy 0, zlib.compress gives 108041 bytes
With compression level 8, mem_level 9, strategy 0, zlib.compress gives 105321 bytes
With compression level 9, mem_level 9, strategy 0, zlib.compress gives 104264 bytes
With compression level 0, mem_level 1, strategy 1, zlib.compress gives 2427294 bytes
With compression level 1, mem_level 1, strategy 1, zlib.compress gives 176182 bytes
With compression level 2, mem_level 1, strategy 1, zlib.compress gives 160285 bytes
With compression level 3, mem_level 1, strategy 1, zlib.compress gives 146284 bytes
With compression level 4, mem_level 1, strategy 1, zlib.compress gives 138646 bytes
With compression level 5, mem_level 1, strategy 1, zlib.compress gives 129359 bytes
With compression level 6, mem_level 1, strategy 1, zlib.compress gives 120436 bytes
With compression level 7, mem_level 1, strategy 1, zlib.compress gives 119583 bytes
With compression level 8, mem_level 1, strategy 1, zlib.compress gives 116264 bytes
With compression level 9, mem_level 1, strategy 1, zlib.compress gives 115021 bytes
With compression level 0, mem_level 2, strategy 1, zlib.compress gives 2415384 bytes
With compression level 1, mem_level 2, strategy 1, zlib.compress gives 170222 bytes
With compression level 2, mem_level 2, strategy 1, zlib.compress gives 155723 bytes
With compression level 3, mem_level 2, strategy 1, zlib.compress gives 143172 bytes
With compression level 4, mem_level 2, strategy 1, zlib.compress gives 134176 bytes
With compression level 5, mem_level 2, strategy 1, zlib.compress gives 126207 bytes
With compression level 6, mem_level 2, strategy 1, zlib.compress gives 118319 bytes
With compression level 7, mem_level 2, strategy 1, zlib.compress gives 117738 bytes
With compression level 8, mem_level 2, strategy 1, zlib.compress gives 114529 bytes
With compression level 9, mem_level 2, strategy 1, zlib.compress gives 113488 bytes
With compression level 0, mem_level 3, strategy 1, zlib.compress gives 2409474 bytes
With compression level 1, mem_level 3, strategy 1, zlib.compress gives 164668 bytes
With compression level 2, mem_level 3, strategy 1, zlib.compress gives 150928 bytes
With compression level 3, mem_level 3, strategy 1, zlib.compress gives 139918 bytes
With compression level 4, mem_level 3, strategy 1, zlib.compress gives 128996 bytes
With compression level 5, mem_level 3, strategy 1, zlib.compress gives 121984 bytes
With compression level 6, mem_level 3, strategy 1, zlib.compress gives 115209 bytes
With compression level 7, mem_level 3, strategy 1, zlib.compress gives 114679 bytes
With compression level 8, mem_level 3, strategy 1, zlib.compress gives 111747 bytes
With compression level 9, mem_level 3, strategy 1, zlib.compress gives 110594 bytes
With compression level 0, mem_level 4, strategy 1, zlib.compress gives 2406529 bytes
With compression level 1, mem_level 4, strategy 1, zlib.compress gives 161203 bytes
With compression level 2, mem_level 4, strategy 1, zlib.compress gives 148187 bytes
With compression level 3, mem_level 4, strategy 1, zlib.compress gives 137649 bytes
With compression level 4, mem_level 4, strategy 1, zlib.compress gives 125905 bytes
With compression level 5, mem_level 4, strategy 1, zlib.compress gives 119155 bytes
With compression level 6, mem_level 4, strategy 1, zlib.compress gives 113051 bytes
With compression level 7, mem_level 4, strategy 1, zlib.compress gives 112575 bytes
With compression level 8, mem_level 4, strategy 1, zlib.compress gives 109749 bytes
With compression level 9, mem_level 4, strategy 1, zlib.compress gives 108727 bytes
With compression level 0, mem_level 5, strategy 1, zlib.compress gives 2405059 bytes
With compression level 1, mem_level 5, strategy 1, zlib.compress gives 159511 bytes
With compression level 2, mem_level 5, strategy 1, zlib.compress gives 146528 bytes
With compression level 3, mem_level 5, strategy 1, zlib.compress gives 136218 bytes
With compression level 4, mem_level 5, strategy 1, zlib.compress gives 124186 bytes
With compression level 5, mem_level 5, strategy 1, zlib.compress gives 117731 bytes
With compression level 6, mem_level 5, strategy 1, zlib.compress gives 111589 bytes
With compression level 7, mem_level 5, strategy 1, zlib.compress gives 111112 bytes
With compression level 8, mem_level 5, strategy 1, zlib.compress gives 108285 bytes
With compression level 9, mem_level 5, strategy 1, zlib.compress gives 107157 bytes
With compression level 0, mem_level 6, strategy 1, zlib.compress gives 2404324 bytes
With compression level 1, mem_level 6, strategy 1, zlib.compress gives 158833 bytes
With compression level 2, mem_level 6, strategy 1, zlib.compress gives 146029 bytes
With compression level 3, mem_level 6, strategy 1, zlib.compress gives 135299 bytes
With compression level 4, mem_level 6, strategy 1, zlib.compress gives 123926 bytes
With compression level 5, mem_level 6, strategy 1, zlib.compress gives 117330 bytes
With compression level 6, mem_level 6, strategy 1, zlib.compress gives 111161 bytes
With compression level 7, mem_level 6, strategy 1, zlib.compress gives 110609 bytes
With compression level 8, mem_level 6, strategy 1, zlib.compress gives 107642 bytes
With compression level 9, mem_level 6, strategy 1, zlib.compress gives 106434 bytes
With compression level 0, mem_level 7, strategy 1, zlib.compress gives 2403959 bytes
With compression level 1, mem_level 7, strategy 1, zlib.compress gives 157836 bytes
With compression level 2, mem_level 7, strategy 1, zlib.compress gives 145142 bytes
With compression level 3, mem_level 7, strategy 1, zlib.compress gives 134822 bytes
With compression level 4, mem_level 7, strategy 1, zlib.compress gives 122917 bytes
With compression level 5, mem_level 7, strategy 1, zlib.compress gives 116738 bytes
With compression level 6, mem_level 7, strategy 1, zlib.compress gives 110681 bytes
With compression level 7, mem_level 7, strategy 1, zlib.compress gives 110313 bytes
With compression level 8, mem_level 7, strategy 1, zlib.compress gives 107484 bytes
With compression level 9, mem_level 7, strategy 1, zlib.compress gives 106377 bytes
With compression level 0, mem_level 8, strategy 1, zlib.compress gives 2403954 bytes
With compression level 1, mem_level 8, strategy 1, zlib.compress gives 157680 bytes
With compression level 2, mem_level 8, strategy 1, zlib.compress gives 145059 bytes
With compression level 3, mem_level 8, strategy 1, zlib.compress gives 134853 bytes
With compression level 4, mem_level 8, strategy 1, zlib.compress gives 123086 bytes
With compression level 5, mem_level 8, strategy 1, zlib.compress gives 116643 bytes
With compression level 6, mem_level 8, strategy 1, zlib.compress gives 110820 bytes
With compression level 7, mem_level 8, strategy 1, zlib.compress gives 110508 bytes
With compression level 8, mem_level 8, strategy 1, zlib.compress gives 107777 bytes
With compression level 9, mem_level 8, strategy 1, zlib.compress gives 106770 bytes
With compression level 0, mem_level 9, strategy 1, zlib.compress gives 2403954 bytes
With compression level 1, mem_level 9, strategy 1, zlib.compress gives 158120 bytes
With compression level 2, mem_level 9, strategy 1, zlib.compress gives 145273 bytes
With compression level 3, mem_level 9, strategy 1, zlib.compress gives 134947 bytes
With compression level 4, mem_level 9, strategy 1, zlib.compress gives 123774 bytes
With compression level 5, mem_level 9, strategy 1, zlib.compress gives 117299 bytes
With compression level 6, mem_level 9, strategy 1, zlib.compress gives 111406 bytes
With compression level 7, mem_level 9, strategy 1, zlib.compress gives 111126 bytes
With compression level 8, mem_level 9, strategy 1, zlib.compress gives 108399 bytes
With compression level 9, mem_level 9, strategy 1, zlib.compress gives 107291 bytes

This means that we can’t reproduce the same compressed data by using the standard implementation of DEFLATE algorithm using the zlib module in Python. Compare the above test results, we can see the following differences between PNG files created by Gimp and Windows Paint on Windows 10:

• IDAT chunks are in 8192-byte blocks in Gimp and 65445-byte blocks in Paint
• Windows Paint on Windows 10 must have applied a kind of pre-compression filtering to optimize the PNG file.
• Windows Paint on Windows 10 doesn’t use the optimal compression parameters.

Further experiments with Gimp 2.8 and Windows Paint on Windows 7 have shown that those applications use the standard DEFLATE algorithm to compress data in PNG files.

Optimize PNG files

Basically, there are three ways to optimize PNG files, namely,

1. Use indexed colors instead of truecolors
2. Filter image data before compression
3. Use different implementations of DEFLATE algorithm which give better compression ratios, ie, 7-zip DEFLATE and Zopfli released by Google

Please refer to the following pages for more information about how to optimize PNG files:

1. https://zoompf.com/blog/2010/01/top-png-optimizers-dont-use-zlib
2. http://www.smashingmagazine.com/2009/07/clever-png-optimization-techniques/

The post Why the Same Screenshot is Saved as PNG of Different Sizes by Different Tools appeared first on Shajisoft.

As in previous Windows 7/8/8.1, if the system locale is set to English, Chinese text would appear ugly in some applications.

Recently, I upgraded to Windows 10 from Window 7. After the Windows 10 was activated, I did a clean installation. Then I had the same issue again. I thought it could be interesting to write a blog (my very first blog at Shajisoft.com) to record how I got rid of the issues.

A picture is worth one thousand words. The following is a screenshot of the main window of TotalCommander with the language switched to Chinese. Those Chinese text appears really ugly. Words are displayed in different heights/sizes respectively as if they were rendered by using different fonts.

The same issue happens to other applications including Windows system tools like Notepad and CMD, as shown below.

Just in case someone is curious, the default language of my Windows 10 box is English, as shown below

As shown above, Chinese text appears normal on the dialog of settings dialog. This means that the ugly display of Chinese text happens to some applications only. In addition, although the issue is with Chinese text, I believe the same issue would happen to Japanese and Korean text as well if the system default language is English.

Testing Environment

All the tests shown in this post were done on Windows 10 English x64 using the following applications:

• TotalCommander 8.52 x64
• CMD (system default)
• Notepad (system default)

Analysis

Basically, the above screenshots show two issues, namely incorrect font or not font found to display Chinese text respectively. Windows makes use of several techniques to find font glyphs to display text. If an incorrect font is mapped, the display of the script would be abnormal. In cmd.exe, text rendering may be different, that is, if no proper font glyphs are found, the text would appear to be boxes. This usually happens to multi-byte text scripts in Chinese, Japanese, and Korean. So, to get around the issue, we need to map the fonts properly when needed.

Solution 1 of 2: Select correct font

If applications like TotalCommander allow users to select fonts, we can select correct font which include the font glyphs needed to display the text, for instance Microsoft Yahei. In TotalCommander, the configuration of font can be modified on the configuration dialog. To open the configuration dialog, select menu Configuration–>Options. The configuration of font is available on “Font” tab page. The default settings of the fonts after the language is switched to Chinese are as shown below:

If we select a proper font, ie, Yahei UI, the Chinese text would be rendered normally, as shown below:

As shown above, once a correct font is select, the problem would be gone. Whereas, this solution is not optimal. In some cases, we still want to have English text rendered using a pretty mono font, while Chinese (or Japanese, Korean) text is still rendered properly, especially in applications like Notepad and CMD. In that case, we may need to set up proper font mapping which will be explained in the following section.

Solution 2 of 2: Set up correct fontlink in the registry

Fontlink is a virtual font technique used in Windows to link many fonts. A font can be linked to many other fonts. In this case, the font is called base font, the other fonts are linked fonts. We can set the application to use a proper base font mainly for English text, and set up the font link to allow proper rendering for texts in other language.

Many pages or posts can be found on Google if we search “Chinese English Windows text”, “fontlink fontlink meiryo ui”, or “fontlink yahei” etc. One can always refer to any of those on how to make changes in the registry on Windows for Fontlink. In this post, I am not going to show the steps in detail, but rather I’ll focus on the discussion on one of the important parameter called “scaling factor”, which is either not covered or explained unclearly in web pages.

Take TotalCommander as the example, the default font is Microsoft San Serif, as shown in the above screenshot. To allow proper rendering for Chinese text, all we need to do is to set up the proper fontlink. Fontlink is defined in Windows registry. To view or edit the registry key, run command “regedit”. Then browse to HKLM–>Software–>Microsoft–>Windows NT–>CurrentVersion–>Fontlink–>SystemLink.

NB: It’s highly recommended to make a backup of the original key value by exporting the original registry key to a file by using the menu File–>Export… in RegEdit.

The above registry key contains all fontlinks for some fonts which includes Microsoft Sans Serif. To view or modify fontlink for Microsoft Sans Serif, righ-click on the entry, and select “Modify…”. The default configuration of the fontlink for Microsoft Sans Serif on Windows 10 English is as shown below:

When looking for font glyphs to render text, Windows checks the list of linked fonts from top to bottom. To allow Chinese text properly rendered, we can add an appropriate font at the top of the list. Before making actual changes, let’s review the syntax of each line in the list defined in the fontlink.

Taking Microsoft Yahei UI as the example, the syntax of an item in the fontlink is as follows:

MSYH.TTC,Microsoft YaHei UI,128,96

The whole line is used by Windows when mapping font. Therefore, the following two lines are two different entries in terms of fontlink:

MSYH.TTC,Microsoft YaHei UI,128,96
MSYH.TTC,Microsoft YaHei UI

where “128,96” are known as scaling factors. According to this link, the above two lines are for GDI and GDI+ respectively. That is, the first line will be picked up by GDI, while the second one is for GDI+. In order to add a new linked font which can be used by both GDI and GDI+, we ought to add two lines in the fontlink registry, one with scaling factors and one without respectively. TotalCommander uses GDI to render text, so we need to set up at least the fontlink with the two integers.

I can’t find any useful information about how the scaling factors are used. But it seems to be true that if the two scaling factors are 128 and 96 respectively, the text won’t be scaled. But in my case, the non-scaled Yahei UI looks a big taller than Micsoft Sans Serif. I have to make the Chinese words appear shorter. Eventually, I set the fontlink as follows for TotalCommander:

NB: To take effect of the new fontlink, sigh out and in Windows again. There is no need to reboot the computer!

With the new fontlink configuration, as shown in the following screenshot, Chinese text rendered properly even though the select font is still Microsoft Sans Serif.

Now, I would like to elaborate a bit more on the scaling factor numbers. It’s mysterious how the numbers are used. It’s quite true that the default values of the two factors are 128 and 96 respectively. In other words, if the two default values, 128,96, are set in fontlink, there won’t be scaling effect. But many web pages would suggest other values, which I’ve found is very confusing, especially some suggestions to set the factors to values by multiplying the default ones with the same number (ie, 1.6). Please read on for the reasons.
I can’t find any useful information about the details of the two numbers. The only one relevant reference is a source file of gdipp project where the two factors are used to compute a new scaling factor as follows

new_info.scaling = (factor1 / 128.0) * (96.0 / factor2);

This means that setting factors by multiplying the default values with the same value doesn’t make any difference! With the above formula and some experiments, I think that we can draw the following conclusions regarding the two scaling factors:

1. To allow programing using GDI to take advantage of Fontlink, we need to add items with scaling factors in the registry.
2. The scaling factors may depend on the actual DPI. In other words, different display device may require different scaling factors in the fontlink registry.
3. The linked fonts are not necessarily in the same size of the base fonts. To increase or reduce the size of the linked font, it’s possible to keep one of the scaling factor to the default value, and just change the other one. It doesn’t matter which one has the default value.
4. Setting scaling factors by multiplying the default values (128 and 96) with the same value doesn’t make any difference.

Still taking TotalCommander as the example, let’s compare the results by setting the scaling factors to the following values respectively:

1. 128,100, and 122, 96 respectively. They should give the same results.
2. 128,128, and 64,64 respectively. They should give the same results.
3. 128,72, and 171,92 respectively. The should give the same results.

The following figure shows the screenshots of TotalCommander with different scaling factors used in fontlink.

On my 20 inch LCD monitor set to 1920×1080, using the default scaling factor, 128 and 96, Chinese text appears a little taller than English text in ToralCommander. I have to set the scaling factor either to “128,100” or “122,96”. The default scaling factors work fine in Notepad and CMD.exe.

Summary

On Windows 10 English, Chinese, Japanese and Korean (CJK) text may be rendered by using a wrong font therefore appear ugly. There are basically two solutions to get rid of the issue, namely to set a correct font in the application or to set correct fontlink in the registry so that a correct font can be used for CJK text.

The second solution is usually preferred mainly for two reasons:

1. We still want English text to be rendered in the preferred font (ie, Cosonlas).
2. The application may not allow us to change the font.

Using fontlink technique, a base font is linked to many fonts. The linked fonts may not necessarily in the same size as the base font. It’s necessary to scale the linked font in this case. It’s mysterious how the scaling factors are used in Windows. But it has been shown that to scale up or down the linked font, we can set one of the scaling factors to the default value, and just change the other one. It doesn’t matter which one has the default value. And multiplying the two default values with the same value doesn’t make a difference.

The post Fontlink for CJK on English Windows 10 appeared first on Shajisoft.