Go language has a strong type system unlike C, and some times this
will be head ache when we want to interact with C data types with Cgo
or just to convert a Go type to lets say byte slice. I recently faced
the same problem and after poking around things, I learned the Go
provides
Solution
Basically C array are sequence of memory location which can be
statically or dynamically allocated. In
Cgo its possible to access C
standard library functions for memory allocation, so why not use
it. The memory allocation function then returns the pointer to
starting of allocated memory, we can use this pointer to write Go's
bytes into the memory location and bytes from memory location into
Go's slice.
The pointer returned by C allocation functions are not directly usable
for memory dereferencing in Go, here is where
unsafe package kicks
in. We will cast the return of C allocation function as
unsafe.Pointer type and from the documentation of unsafe package,
- A pointer value of any type can be converted to a Pointer.
- A Pointer can be converted to a pointer value of any type.
- A uintptr can be converted to a Pointer.
- A Pointer can be converted to a uintptr.
So we can then cast unsafe.Pointer to
uintptr which is the Go type
which is large enough to hold any memory address in Go and can be used
for pointer arithmetic just like we do in C (of course with some more
castings).
Below I'm pasting a simplified code in C which I wrote for this
post.
#ifndef __BYTETEST_H__
#define __BYTETEST_H__
typedef unsigned char UBYTE;
extern void ArrayReadFunc(UBYTE *arrayout);
extern void ArrayWriteFunc(UBYTE *arrayin);
#endif
#include "bytetest.h"
#include <stdio.h>
#include <string.h>
void ArrayReadFunc(UBYTE *arrayout)
UBYTE array[20] = 1, 2, 3, 4,5, 6, 7, 8, 9, 10,
11, 12, 13, 14, 15, 16, 17,
18, 19, 20 ;
memcpy(arrayout, array, 20);
void ArrayWriteFunc(UBYTE *arrayin)
UBYTE array[20];
memcpy(array, arrayin, 20);
printf("Byte slice array received from Go:\n");
for(int i = 0; i < 20; i ++)
printf("%d ", array[i]);
printf("\n");
Functions are written just for this post and they don't really do
anything. As you can see
ArrayReadFunc takes a pointer to array and
fills it with content of another array using
memcpy. Function
ArrayWriteFunc on other hand takes pointer to array and copies its
content to internal array. I've added print logic to
ArrayWriteFunc
just to show that values passed from
Go are making it here.
Below is the Go code which uses the above C files passes byte slice to
get value out of C code and array made of byte slice to C function to
send values in.
package main
/*
#cgo CFLAGS: -std=c99
#include "bytetest.h"
#include <stdlib.h>
*/
import "C"
import (
"fmt"
"unsafe"
)
func ReadArray() unsafe.Pointer
var outArray = unsafe.Pointer (C.calloc(20,1))
C.ArrayReadFunc((*C.UBYTE)(outArray))
return outArray
func WriteArray(inArray unsafe.Pointer)
C.ArrayWriteFunc((*C.UBYTE)(inArray))
func CArrayToByteSlice(array unsafe.Pointer, size int) []byte
var arrayptr = uintptr(array)
var byteSlice = make([]byte, size)
for i := 0; i < len(byteSlice); i ++
byteSlice[i] = byte(*(*C.UBYTE)(unsafe.Pointer(arrayptr)))
arrayptr ++
return byteSlice
func ByteSliceToCArray (byteSlice []byte) unsafe.Pointer
var array = unsafe.Pointer(C.calloc(C.size_t(len(byteSlice)), 1))
var arrayptr = uintptr(array)
for i := 0; i < len(byteSlice); i ++
*(*C.UBYTE)(unsafe.Pointer(arrayptr)) = C.UBYTE(byteSlice[i])
arrayptr ++
return array
func main()
carray := ReadArray()
defer C.free(carray)
carraybytes := CArrayToByteSlice(carray, 20)
fmt.Println("C array converted to byte slice:")
for i := 0; i < len(carraybytes); i ++
fmt.Printf("%d ", carraybytes[i])
fmt.Println()
gobytes := []byte 21, 22, 23, 24, 25, 26, 27, 28, 29, 30,
31, 32, 33, 34, 35, 36, 37, 38, 39, 40
gobytesarray := ByteSliceToCArray(gobytes)
defer C.free(gobytesarray)
WriteArray(gobytesarray)
Functions
ReadArray and
WriteArray are just wrapper to the calls
to C counter parts
ArrayReadFunc and
ArrayWriteFunc.
ReadArray
returns
unsafe.Pointer which is allocated C array and should be
freed by caller.
WriteArray takes
unsafe.Pointer which is pointing
to memory location containing C array.
Now the functions of interest are
CArrayToByteSlice and
ByteSliceToCArray. It should be pretty clear from the above code to
understand what is happening in these functions. Still I will just put
explain them briefly.
ByteSliceToCArray allocates a C array using
calloc from C standard
library. It then creates a
uintptr, a pointer type in Go which is
used to dereference the each memory location and store bytes from the
input byte slice in them.
CArrayToByteSlice on other hands creates a
uintptr type by casting
input unsafe.Pointer type and then uses this pointer type to
dereference values from memoy and store it in byte slice with suitable
casting.
So lets build the code and run it and see the output:
C array converted to byte slice:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Byte slice array received from Go:
21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
So yes it actually works and values are moving across C and Go. This
solves first problem in hand next is converting arbitrary Go types
into byte slices.
There are many cases where we would like to convert an arbitrary Go
types like (int, float) into bytes. One such use case I found was when
writing a TCP client for communicating with a Server written using C
speaking custom protocol. Here I'm just going to show how to convert
types like float, int to byte slice, I've not tried converting
structures but it is certainly possible.
Below is the function which can convert int32,float32 into byte slice
it can also be extended for other types.
func CopyValueToByte(value interface ) []byte
var valptr uintptr
var slice []byte
switch t := value.(type)
case int32:
i := value.(int32)
valptr = uintptr(unsafe.Pointer(&i))
slice = make([]byte, unsafe.Sizeof(i))
case float32:
f := value.(float32)
valptr = uintptr(unsafe.Pointer(&f))
slice = make([]byte, unsafe.Sizeof(f))
default:
fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
os.Exit(1)
for i := 0; i < len(slice); i++
slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
valptr++
return slice
This function is generic which can take various types of value. First
we will use Go's type assertion to determine the type and creates a
uintptr pointer for the value and allocates byte slice depending on
the size of the value as calculated using
unsafe.Sizeof. Later it
uses the pointer to dereference value from memory location and copies
each byte into byte slice.
The idea used here is every type is
represented as certain number of bytes in the memory. Below is the
entire program.
package main
import (
"fmt"
"unsafe"
"os"
)
func CopyValueToByte(value interface ) []byte
var valptr uintptr
var slice []byte
switch t := value.(type)
case int32:
i := value.(int32)
valptr = uintptr(unsafe.Pointer(&i))
slice = make([]byte, unsafe.Sizeof(i))
case float32:
f := value.(float32)
valptr = uintptr(unsafe.Pointer(&f))
slice = make([]byte, unsafe.Sizeof(f))
default:
fmt.Fprintf(os.Stderr,"Unsupported type: %T\n", t)
os.Exit(1)
for i := 0; i < len(slice); i++
slice[i] = byte(*(*byte)(unsafe.Pointer(valptr)))
valptr++
return slice
func main()
a := float32(-10.3)
floatbytes := CopyValueToByte(a)
fmt.Println("Float value as byte slice:")
for i := 0; i < len(floatbytes); i++
fmt.Printf("%x ", floatbytes[i])
fmt.Println()
b := new(float32)
bptr := uintptr(unsafe.Pointer(b))
for i := 0; i < len(floatbytes); i++
*(*byte)(unsafe.Pointer(bptr)) = floatbytes[i]
bptr++
fmt.Printf("Byte value copied to float var: %f\n", *b)
The above conversion can also be achieved using
encoding/binary
package provided by Go. But
its been told to me that it
makes things pretty slow.
Recently I've been seeing lot of posts about DNSSEC on Internet and I
thought I should configure my domain to be secured by DNSSEC.
copyninja.info domain is now secured with DNSSEC you can verify
this by DNSSEC analyzer by Verisign and DNSViz online tool or by
installing DNSSEC validator addon for
your browser.
There are good amount of tutorials and guides available to enable
DNSSEC for your domain, still I want to note down steps I followed to
here for the record (of course it will be helpful for me if I forget
it ;-))
First step will be installing bind9 and dnssec-tools package, if
you use aptitude installing dnssec-tools will pull down the bind9
unless you have configured aptitude to not install the Recommends.
Next setting up the zone file for your domain, for this first make a
copy of /etc/bind/db.local as /etc/bind/db.example.com, replace
example.com with your domain name. Now you need to add your zone
records to the zone file.
Next edit the /etc/bind/named.conf.local file and add following lines
So you can see no response header in the above image but there is a
response and below how is response content visible on firebug.
Still weird part was when using wget/curl directly to get the CSS I
was getting correct reply and the file, which is puzzling.
So I went ahead opened the CSS url directly using browser and saved
the resulting file. When I used file command on it, output
suggested the saved file in gzip compressed!. I went ahead
uncompressed it and opened the file and inside it I found response
header and the response body! So what just happened? Did apache some
how manage to compress entire respone not just response body?
The more puzzling question was why did it work when I was using
mod_proxy_uwsgi and failed when I switched to mod_uwsgi. I had no
clue at this point why this behavior is coming up.
So response is still gziped but this time only response body was
gziped and Content-Encoding field was set properly which makes browser
to properly uncompress the response body and use it.
So it seems like there is a difference between mod_uwsgi and
mod_proxy_uwsgi implementation. mod_uwsgi was sending the response
along with response headers which was compressed by mod_deflate there
by rendering browser helpless to interpret the response. But
mod_proxy_uwsgi seems to be only sending the response content without
response body and this content was later compressed by mod_deflate and
proper response headers were set by apache before sending it to
browser.
So now whose fault is this? Is it the bug in mod_uwsgi or, mod_deflate
not being able to exclude response headers from compression? I've no
clue! If you have a clue and you want to share it with me, please
consider writing to me over email. My details are available in
Good bye Sir :-(