Adding to Secp256k1 JNI
Bitcoin-S uses a Java Native Interface (JNI) to execute functions in secp256k1 from java/scala. The native java bindings used to be a part of the secp256k1 library that was maintained by bitcoin-core, but it was removed in October 2019. We maintain a fork of secp256k1 which forks off of bitcoin-core's master
but re-introduces the jni. This is also the easiest way to add functionality from new projects such as Schnorr signatures and ECDSA adaptor signatures by rebasing the bitcoin-s branch with the JNI on top of these experimental branches. That said, it is quite tricky to hook up new functionality in secp256k1 into bitcoin-s and specifically NativeSecp256k1.java
. The following is a description of this process.
Adding a new function to NativeSecp256k1.java
src/java/org_bitcoin_NativeSecp256k1.c
Adding to Add an
#include
import at the top (if applicable)If your secp256k1 functions are not already included, you will need to
#include
the header file (should be in thesecp256k1/include
directory).Function signature
Your new function signature should begin with
SECP256K1_API jint JNICALL Java_org_bitcoin_NativeSecp256k1_
followed by the secp256k1 function name where
_
are replaced with_1
(it's a weird jni thing). Finally, you add a parameter list that begins with(JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l
and ends with any
jint
s in the case that any of the secp256k1 function inputs have variable length (such as public keys which can be either33
or65
bytes, or ECDSA signatures), and lastly anyjboolean
s in case there is some flag likecompressed
passed in.As an example that includes everything, if you are making a call to
secp256k1_pubkey_tweak_add
which takes in public keys that could be33
or65
bytes and outputs a public key that will either be compressed or decompressed based on an input flag, then the function signature would beSECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add (JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l, jint publen, jboolean compressed)
Reading
unsigned char*
inputsIt is now time to create pointers for each of the secp256k1 function inputs that where passed in via the
byteBufferObject
. We must first read in theSecp256k1Context
with the linesecp256k1_context *ctx = (secp256k1_context*)(uintptr_t)ctx_l;
and we can then initialize the first pointer to be the beginning of the
byteBufferObject
with the lineunsigned char* firstArg = (*env)->GetDirectBufferAddress(env, byteBufferObject);
and subsequent arguments' pointers where the previous argument's length is known (say
32
bytes for example) can be instantiated usingunsigned char* prevArg = ... unsigned char* nextArg = (unsigned char*) (prevArg + 32);
and in the case that a previous argument has variable length, then a
jint
has been provided as an input and can be used instead, such as in the exampleunsigned char* prevArg = ... unsigned char* nextArg = (unsigned char*) (prevArg + publen);
where
publen
is ajint
passed to this C function.As an example that includes everything, consider the function
secp256k1_ecdsa_verify
which takes as input a32
byte message, a variable length signature and a public key (of length33
or65
bytes). Our function will begin with{ secp256k1_context *ctx = (secp256k1_context*)(uintptr_t)ctx_l; unsigned char* data = (unsigned char*) (*env)->GetDirectBufferAddress(env, byteBufferObject); const unsigned char* sigdata = (unsigned char*) (data + 32); const unsigned char* pubdata = (unsigned char*) (sigdata + siglen);
where
siglen
is ajint
passed into the C function.Initialize variables
Next we must declare all variables. We put all decelerations here as it is required by the C framework used by
libsecp256k1
that definitions and assignments/function calls cannot be interleaved.Specifically you will need to declare any secp256k1 specific structs here as well as all outputs (such as
jobjectArrays
andjByteArrays
). Generally speaking this will include all inputs which are not raw data (public keys, signatures, etc). Lastly, you will also have anint ret
which will store0
if an error occurred and1
otherwise.As an example that includes everything, consider again the function
secp256k1_pubkey_tweak_add
has the following declarationsjobjectArray retArray; jbyteArray pubArray, intsByteArray; unsigned char intsarray[2]; unsigned char outputSer[65]; size_t outputLen = 65; secp256k1_pubkey pubkey; int ret;
Where
retArray
is eventually going to be the data returned, which will contain thejbyteArray
spubArray
andintsByteArray
, which will containoutputSer
andintsarray
respectively. Lastlypubkey
will store a deserializedsecp256k1_pubkey
corresponding to the inputunsigned char*
public key.Parse inputs when applicable
In the case where there are
unsigned char*
inputs which need to be deserialized into secp256k1 structs, this is done now. As an example,secp256k1_pubkey_tweak_add
takes a public key as input:unsigned char* pkey = (*env)->GetDirectBufferAddress(env, byteBufferObject);
where a
jint publen
is passed in as a function parameter. This function already has a declaration forsecp256k1_pubkey pubkey;
. The first call made after the above declarations isret = secp256k1_ec_pubkey_parse(ctx, &pubkey, pkey, publen)
and if further parsing is necessary, it is put inside of
if (ret) { ret = [further parsing here] }
.Make calls to secp256k1 functions to instantiate outputs
It is finally time to actually call the secp256k1 function we are binding to the jni! This is done by simply calling
ret = [call to secp function here];
orif (ret) { ret = [secp function call] };
if there were any inputs that needed to be parsed. Note that some secp256k1 functions return outputs by populating variables you should have declared and for which pointers are passed as inputs, while other functions will mutate their inputs rather than returning outputs.Serialize variable length outputs if applicable
When dealing with variable length outputs such as signatures, you will likely need to serialize these outputs. This is done by having already instantiated such a variable as
unsigned char outputSer[72]; size_t outputLen = 72;
where in this case
72
is an upper bound on signature length. With these variables existing (as well as asecp256k1_ecdsa_signature sig
which has been populated), we call a secp256k1 serialization function to populateoutputSer
andoutputLen
fromsig
if(ret) { int ret2 = secp256k1_ecdsa_signature_serialize_der(ctx, outputSer, &outputLen, &sig); (void)ret2; }
As you can see, in this case we do not which to alter the value returned in
ret
if serialization fails. If we did thenret2
would not be introduced and we would instead doret = [serialize]
.Populate return array when applicable
We now begin translating our serialized results back into Java entities. If you are returning any
int
s containing meta-data (usuallyret
is included here, as are the variable lengths of outputs when applicable), you will want anunsigned char intsarray[n]
to be already declared wheren
is the number of pieces of meta-data. For example, insecp256k1_ecdsa_sign
, we wish to return whether there were any errors (stored inint ret
) and the output signature's length,size_t outputLen
. Hence we have anunsigned char intsarray[2]
and we populate it as followsintsarray[0] = outputLen; intsarray[1] = ret;
Next we populate the
jobjectArray
we wish to return, this will always begin with a callretArray = (*env)->NewObjectArray(env, 2, (*env)->FindClass(env, "[B"), (*env)->NewByteArray(env, 1));
to instantiate an empty return array. Next we instantiate our
jbyteArray
s with calls tomyByteArray = (*env)->NewByteArray(env, len);
where
myByteArray
is replaced with a real name (such asintsByteArray
) andlen
is replaced with the length of this array (either a constant or a populatedsize_t
variable). Next we populate this array with our data by calling(*env)->SetByteArrayRegion(env, myByteArray, 0, len, (jbyte*)myData);
where
myData
is a C array ofunsigned char
(of lengthlen
). Lastly, we placemyByteArray
into its place inretArray
with(*env)->SetObjectArrayElement(env, retArray, index, myByteArray);
where
index
is a constant (0
,1
,2
, etc.) for the index ofmyByteArray
withinretArray
. Note that you should follow our conventions and have index0
contain the actual data to be returned and index1
(and onward) contain any meta-data.Please note that if you wish not to return such meta-data (such as if you wish to return only a
boolean
), then none of the code in this subsection is requiredvoid
classObject
Once we are ready to return, we first void the input
classObject
by making the call(void)classObject;
Return array when applicable,
ret
when applicableLastly, we return
retArray
in the case where we wish to return abyte[]
, orret
in the case that we wish to return aboolean
.
src/java/org_bitcoin_NativeSecp256k1.h
Adding to Once your function is defined in src/java/org_bitcoin_NativeSecp256k1.c
, you must define them in the corresponding header files by simply copying the function signature but without parameter names. For example, if secp256k1_pubkey_tweak_add
has the function signature
SECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add
(JNIEnv* env, jclass classObject, jobject byteBufferObject, jlong ctx_l, jint publen, jboolean compressed)
then in the header file we include
/*
* Class: org_bitcoin_NativeSecp256k1
* Method: secp256k1_pubkey_tweak_add
* Signature: (Ljava/nio/ByteBuffer;JI)[[B
*/
SECP256K1_API jobjectArray JNICALL Java_org_bitcoin_NativeSecp256k1_secp256k1_1pubkey_1tweak_1add
(JNIEnv *, jclass, jobject, jlong, jint, jboolean);
src/java/org/bitcoin/NativeSecp256k1.java
Adding to We are now done writing C code! We have completed an interface in C for the JNI to hook up to. However, we must now write the corresponding Java code which hides the Java to C (and back) conversions from other Java code. We accomplish this with a class
of static
methods called NativeSecp256k1
.
Add
private static native
secp256k1 functionWe begin by adding a
private static native
method at the bottom of the file corresponding to our secp256k1 function. Notice that the syntax fornative
methods is similar to that of Java abstract interface methods where instead of providing an implementation we simply end with a semi-colon.For functions returning
boolean
s, we have theirnative
methods returnint
(will be0
or1
). Otherwise, for functions returningbyte[]
s, we have theirnative
methods returnbyte[][]
(two dimensional array to allow for meta-data).Method signature
Next we add a method to the
NativeSecp256k1
classpublic static byte[] myFunc(byte[] input1, byte[] input2, boolean input3) throws AssertFailException
where
boolean
could also be the return type instead ofbyte[]
.checkArgument
sWe begin implementing this function by checking the input argument lengths using the
checkArument
functioncheckArgument(input1.length == 32 && (input2.length == 33 || input2.length == 65));
Initialize
ByteBuffer
We now initialize the
ByteBuffer
which we will be passing through the JNI as an input. This is done with a call toByteBuffer byteBuff = nativeECDSABuffer.get();
followed by allocation when necessary as follows
if (byteBuff == null || byteBuff.capacity() < input1.length + input2.length) { byteBuff = ByteBuffer.allocateDirect(input1.length + input2.length); byteBuff.order(ByteOrder.nativeOrder()); nativeECDSABuffer.set(byteBuff); }
where
input1.length + input2.length
is replaced by whatever the totalByteBuffer
length needed.Fill
ByteBuffer
We now populate the
ByteBuffer
as followsbyteBuff.rewind(); byteBuff.put(input1); byteBuff.put(input2);
where generally, you will
rewind()
and thenput()
all inputs (in order).Make
native
callIt is now time to make a call to our
native
C function.In the case where we are returning a
byte[]
, this is done by first declaring abyte[][]
to store the output and then locking the read lock,r
. Then we call thenative
function within atry
clause which releases the lock in thefinally
clause.byte[][] retByteArray; r.lock(); try { retByteArray = secp256k1_my_call(byteBuff, Secp256k1Context.getContext(), input3); } finally { r.unlock(); }
In the case where we are returning a
boolean
, simply make the call in thetry
and compare the output to1
like sor.lock(); try { return secp256k1_my_bool_call(byteBuff, Secp256k1Context.getContext()) == 1; } finally { r.unlock(); }
If this is the case, you are now done and can ignore the following steps.
Parse outputs
retByteArray
should now be populated and we want to read its two parts (data and meta-data). Getting the data should be as easy asbyte[] resultArr = retByteArr[0];
while for each piece of meta-data, you can read the corresponding
int
as followsint metaVal = new BigInteger(new byte[] { retByteArray[1][index] }).intValue();
where
index
is replaced with the index in the meta-data array.Validate outputs
In the case where we now have meta-data, we validate it with calls to
assertEquals
.Return output
Finally, we return
resultArr
.
src/java/org/bitcoin/NativeSecp256k1Test.java
Adding to I normally first build the C binaries and add to Bitcoin-S before coming back to this section because I use sbt core/console
to generate values and make calls below, but this is not a requirement.
Generate values and make calls to
org.bitcoin.NativeSecp256k1
to generate inputs and their expected outputsCreate regression unit tests with these values in NativeSecp256k1Test
Note that you can use
DatatypeConverter.parseHexBinary
to convertString
hex to abyte[]
, and you can useDatatypeConverter.printHexBinary
to convert abyte[]
to itsString
hex. Lastly you will make assertions with calls toassertEquals
.Add test to
main
Adding to Bitcoin-S
Translate
NativeSecp256k1
andNativeSecp256k1Test
to jni projectBy translate I mean to say that you must copy the functions from those files to the corresponding files in the
bitcoin-s/secp256k1jni
project. For tests this will require changing calls toDatatypeConverter
to eithertoByteArray
ortoHex
as well as changing the method to make it non-static
as well as adding the@Test
annotation above the method (rather than adding to amain
method).Configure and build
secp256k1
You will need to go to the
bitcoin-s/secp256k1
directory in a terminal and running the following where you may need to add to the./configure
command if you are introducing a new module.For Linux or OSx (64-bit)
You will have to make sure
JAVA_HOME
is set, and build tools are installed, for Linux this requires:echo JAVA_HOME sudo apt install build-essential autotools-dev libtool automake
and for Mac this requires:
brew install automake libtool
You should then be able to build
libsecp256k1
with the following:./autogen.sh ./configure --enable-jni --enable-experimental --enable-module-ecdh make make check make check-java
For Windows (64-bit)
Windows bindings are cross-built on Linux. You need to install the
mingw
toolchain and haveJAVA_HOME
point to a Windows JDK:sudo apt install g++-mingw-w64-x86-64 sudo update-alternatives --config x86_64-w64-mingw32-g++
You should then be able to build
libsecp256k1
with the following:echo "LDFLAGS = -no-undefined" >> Makefile.am ./configure --host=x86_64-w64-mingw32 --enable-experimental --enable-module_ecdh --enable-jni && make clean && make CFLAGS="-std=c99"
There may be some errors that can be ignored:
Could not determine the host path corresponding to
redeclared without dllimport attribute: previous dllimport ignored
Copy binaries into bitcoin-s natives for your system
You have now built the C binaries for your JNI bindings for your operating system and you should now find your operating system's directory in
bitcoin-s/secp256k1jni/natives
and replace its contents with the contents ofsecp256k1/.libs
(which contains the compiled binaries).Run
secp256k1jni
testsIf you have not yet implemented tests, you should now be able to go back and do so as calls to
NativeSecp256k1
should now succeed.Once you have tests implemented, and assuming you've copied them correctly to the
bitcoin-s/secp256k1jni
project, you should be able to run them usingsbt secp256k1jni/test
Further Work to Enable Typed Invocations and Nice Tests
Add new
NetworkElement
s where applicableIn the case where you are dealing in new kinds of data that are not yet defined in Bitcoin-S, you should add these as
case class
es extending theNetworkElement
trait, and give them companion objects extendingFactory
for easy serialization and deserialization.This step is not necessary if you are only dealing in raw data,
ECPrivateKey
s,ECPublicKey
s, etc.Add new typed functions to relevant data types where applicable
In the case where your new function should be a static method, find a good
object
(or introduce one) and give it adef
which takes in typed arguments and outputs typed arguments (usingByteVector
in all places dealing with raw data rather than usingArray[Byte]
). You will then implement these methods using calls toNativeSecp256k1
methods and getting the inputs intoArray[Byte]
form by getting theirByteVector
s (usually through a call to_.bytes
) and then calling_.toArray
.You will then need to take the data returned and deserialize it.
In the case where your new function belongs naturally as an action performed by some existing or newly introduced type, you can implement your new function as a call made by that class as described for the previous case but where the class will pass a serialized version of itself into the
NativeSecp256k1
call.It is often acceptable to implement the call in an
object
and then also add the call (via a call to the object, passingthis
) to the interface of relevant types.Implement Bouncy Castle fallback in
BouncyCastleUtil.scala
if you can.Add unit and property-based tests.
If you implemented Bouncy Castle fallback, add tests to
BouncyCastleSecp256k1Test
to compare implementations