[HG DOLFIN] Get typemap(in) for std::vector working.

Discussion:

dolfin-dDist1jlEr4dnm+ (DOLFIN)

2009-11-16 21:03:13 UTC

One or more new changesets pushed to the primary dolfin repository.
A short summary of the last three changesets is included below.

changeset: 7459:739b7539d6eb
tag: tip
user: "Garth N. Wells <gnw20-KWPb1pKIrIJaa/***@public.gmane.org>"
date: Mon Nov 16 21:02:54 2009 +0000
files: dolfin/swig/numpy_typemaps.i dolfin/swig/std_vector_typemaps.i site-packages/dolfin/expression.py
description:
Get typemap(in) for std::vector working.

Seems to work but I don't know if it's any good.

changeset: 7458:f9d6aeab7271
user: "Garth N. Wells <gnw20-KWPb1pKIrIJaa/***@public.gmane.org>"
date: Mon Nov 16 19:06:12 2009 +0000
files: demo/function/eval/python/demo.py site-packages/dolfin/expression.py
description:
Comment out some dim checks in expression.py.

changeset: 7457:6dc505f97225
user: "Garth N. Wells <gnw20-KWPb1pKIrIJaa/***@public.gmane.org>"
date: Mon Nov 16 15:13:08 2009 +0000
files: demo/function/eval/python/demo.py dolfin/swig/docstrings.i dolfin/swig/function_post.i dolfin/swig/function_pre.i dolfin/swig/std_vector_typemaps.i
description:
Add empty typemap for NumPy array to std::vector<double>.

----------------------------------------------------------------------
For more details, visit http://www.fenics.org/hg/dolfin

Johan Hake

2009-11-18 16:23:47 UTC

Permalink

Post by dolfin-dDist1jlEr4dnm+ (DOLFIN)
One or more new changesets pushed to the primary dolfin repository.
A short summary of the last three changesets is included below.
changeset: 7459:739b7539d6eb
tag: tip
date: Mon Nov 16 21:02:54 2009 +0000
files: dolfin/swig/numpy_typemaps.i dolfin/swig/std_vector_typemaps.i
Get typemap(in) for std::vector working.
Seems to work but I don't know if it's any good.

(Resending the email as I forgot to send it using the correct email adress...)

I haven't got the time to look at it, but yes the provided typemaps do make a
copy, which is not optimal. I have have actually been a bit worried ;) that
you might change the interface of eval, and possible other methods too, from
double* to std::vector. In most perspectives it is much nicer to pass a
std::vector, not least for safety. However std::vector is more difficult to
interact with NumPy.

SWIG do provide it's own implementation of std::vector which more or less wrap
the c++ interface of a vector seamless. We have used this to pass arguments of
std:vector previously. However the amount of code that is produced and not
used is large (~1MB for DOLFIN version < 9.2). NumPy arrays and lists are also
much more convenient structures to work with in Python.

I have therefore decided to remove the dependency of std_vector.i by adding
our own typemaps for std::vector arguments. Each one of those typemaps do make
a copy. For now we have typmaps for primitives and pointers so each copy
should not be that bad. But for operations that is performed alot of times, it
might be a difference. But I doubt it would for the callback functionality, as
I assume just calling the python function would take most of the time.

We could consider creating a NumPy array from the pointer to the first element
of the vector, as I have seen you have discussed on the list previously, for
other cases. We then have to consider the life time of the vector. For
director typemaps that would probably be ok. For other "out" typemaps it might
not be ok.

Would it be an option to add, in C++, a small template wrapper class for
arrays that are supposed to go back and forth to Python via NumPy arrays?
Including basic operations so it will be useful also in C++.

Johan

Post by dolfin-dDist1jlEr4dnm+ (DOLFIN)
changeset: 7458:f9d6aeab7271
date: Mon Nov 16 19:06:12 2009 +0000
files: demo/function/eval/python/demo.py
Comment out some dim checks in expression.py.
changeset: 7457:6dc505f97225
date: Mon Nov 16 15:13:08 2009 +0000
files: demo/function/eval/python/demo.py dolfin/swig/docstrings.i
dolfin/swig/function_post.i dolfin/swig/function_pre.i
Add empty typemap for NumPy array to std::vector<double>.
----------------------------------------------------------------------
For more details, visit http://www.fenics.org/hg/dolfin
_______________________________________________
DOLFIN-dev mailing list
http://www.fenics.org/mailman/listinfo/dolfin-dev

Anders Logg

2009-11-18 16:54:22 UTC

Permalink

Post by Johan Hake

(Resending the email as I forgot to send it using the correct email adress...)
I haven't got the time to look at it, but yes the provided typemaps do make a
copy, which is not optimal. I have have actually been a bit worried ;) that
you might change the interface of eval, and possible other methods too, from
double* to std::vector. In most perspectives it is much nicer to pass a
std::vector, not least for safety. However std::vector is more difficult to
interact with NumPy.
SWIG do provide it's own implementation of std::vector which more or less wrap
the c++ interface of a vector seamless. We have used this to pass arguments of
std:vector previously. However the amount of code that is produced and not
used is large (~1MB for DOLFIN version < 9.2). NumPy arrays and lists are also
much more convenient structures to work with in Python.
I have therefore decided to remove the dependency of std_vector.i by adding
our own typemaps for std::vector arguments. Each one of those typemaps do make
a copy. For now we have typmaps for primitives and pointers so each copy
should not be that bad. But for operations that is performed alot of times, it
might be a difference. But I doubt it would for the callback functionality, as
I assume just calling the python function would take most of the time.
We could consider creating a NumPy array from the pointer to the first element
of the vector, as I have seen you have discussed on the list previously, for
other cases. We then have to consider the life time of the vector. For
director typemaps that would probably be ok. For other "out" typemaps it might
not be ok.
Would it be an option to add, in C++, a small template wrapper class for
arrays that are supposed to go back and forth to Python via NumPy arrays?
Including basic operations so it will be useful also in C++.

We have had such a class before named simple_array but it was removed
at some point.

--
Anders

Post by Johan Hake
Johan

_______________________________________________
DOLFIN-dev mailing list
http://www.fenics.org/mailman/listinfo/dolfin-dev

Garth N. Wells

2009-11-18 17:03:16 UTC

Permalink

Post by Johan Hake

double* is easier, but my impression is that it's not particularly safe.

Post by Johan Hake
SWIG do provide it's own implementation of std::vector which more or less wrap
the c++ interface of a vector seamless. We have used this to pass arguments of
std:vector previously. However the amount of code that is produced and not
used is large (~1MB for DOLFIN version < 9.2). NumPy arrays and lists are also
much more convenient structures to work with in Python.

Does the SWIG implementation of std::vector involve copying? I expect
that it does because I don't see how one can create a
std::vector<double> from a list since that would require creating a
std::vector that points to someone else's (the list's) data. It would
break the std::vector design.

I see how it could work if one creates a DoubleVector rather than a
list, but that's ugly.

Post by Johan Hake
I have therefore decided to remove the dependency of std_vector.i by adding
our own typemaps for std::vector arguments. Each one of those typemaps do make
a copy. For now we have typmaps for primitives and pointers so each copy
should not be that bad. But for operations that is performed alot of times, it
might be a difference. But I doubt it would for the callback functionality, as
I assume just calling the python function would take most of the time.

Agree.

Post by Johan Hake
We could consider creating a NumPy array from the pointer to the first element
of the vector, as I have seen you have discussed on the list previously, for
other cases. We then have to consider the life time of the vector.

We need a std::vector that stores a shared_ptr to its data ;).

Post by Johan Hake
For
director typemaps that would probably be ok. For other "out" typemaps it might
not be ok.

I would be inclined to make a copy. It's safe and we can document it so
if one sends a NumPy array into a function that takes a
std::vector<double>&, and that function stores a reference to the
object, then it 's not the original NumPy array (confusing???).

Do we presently copy data between NumPy arrays and the sub-classes of
GenericVector?

Post by Johan Hake
Would it be an option to add, in C++, a small template wrapper class for
arrays that are supposed to go back and forth to Python via NumPy arrays?

Maybe. I see that this is what Hans Petter does in a chapter of one of
his books. Still, there are issues with ownership.

I guess in the end that memory management + pointers is always going to
be tough in a mixed-language environment.

Garth

Post by Johan Hake
Including basic operations so it will be useful also in C++.
Johan

_______________________________________________
DOLFIN-dev mailing list
http://www.fenics.org/mailman/listinfo/dolfin-dev

Johan Hake

2009-11-18 20:38:54 UTC

Permalink

Post by Garth N. Wells

Post by Johan Hake

Post by dolfin-dDist1jlEr4dnm+ (DOLFIN)
One or more new changesets pushed to the primary dolfin repository.
A short summary of the last three changesets is included below.
changeset: 7459:739b7539d6eb
tag: tip
date: Mon Nov 16 21:02:54 2009 +0000
files: dolfin/swig/numpy_typemaps.i
dolfin/swig/std_vector_typemaps.i site-packages/dolfin/expression.py
Get typemap(in) for std::vector working.
Seems to work but I don't know if it's any good.

double* is easier, but my impression is that it's not particularly safe.

I agree. Getting rid of the occasional segfaults that are tricky to debug, and
certainly not understandable for pure Python users, is tractable.

Post by Garth N. Wells

Post by Johan Hake
SWIG do provide it's own implementation of std::vector which more or less
wrap the c++ interface of a vector seamless. We have used this to pass
arguments of std:vector previously. However the amount of code that is
produced and not used is large (~1MB for DOLFIN version < 9.2). NumPy
arrays and lists are also much more convenient structures to work with in
Python.

I actually think it does not involve copying. They have reimplemented the
whole std::vector interface, and integrated it into SWIG type system. That
goes for many of the containers in the standard library. How the communication
with the original C++ library is done is beyond me.

Post by Garth N. Wells
I see how it could work if one creates a DoubleVector rather than a
list, but that's ugly.

I agree. And I have put quite a lot of effort to remove the dependency of
std_vector.i. But if we decide that this is a show stopper we can revert to
std_vector.i. I think we then need to do that for all std::vector typemaps,
including the pointer typemaps for DirichletBC, GenericFunction, FunctionSpace
aso.

Post by Garth N. Wells

Post by Johan Hake
I have therefore decided to remove the dependency of std_vector.i by
adding our own typemaps for std::vector arguments. Each one of those
typemaps do make a copy. For now we have typmaps for primitives and
pointers so each copy should not be that bad. But for operations that is
performed alot of times, it might be a difference. But I doubt it would
for the callback functionality, as I assume just calling the python
function would take most of the time.

Agree.

Ok.

Post by Garth N. Wells

Post by Johan Hake
We could consider creating a NumPy array from the pointer to the first
element of the vector, as I have seen you have discussed on the list
previously, for other cases. We then have to consider the life time of
the vector.

We need a std::vector that stores a shared_ptr to its data ;).

Yeah, that would be nice.

Post by Garth N. Wells

Post by Johan Hake
For
director typemaps that would probably be ok. For other "out" typemaps it
might not be ok.

Thats ok, for all pure in typemaps, for example the coordinates. However when
one expects an out argument, like the values in an eval method, we need to
change any provided values. This could be done in a layer on top of the
typemap. Then we could make the output NumPy array optional. Like we now have
in the __call__ method.

Post by Garth N. Wells
Do we presently copy data between NumPy arrays and the sub-classes of
GenericVector?

For the foo.array() method, yes. But for MTL4 and uBLAS backend we provide a
NumPy array with a pointer to the original data, through the foo.data()
method. This is consistent with the C++ interface.

Post by Garth N. Wells

Post by Johan Hake
Would it be an option to add, in C++, a small template wrapper class for
arrays that are supposed to go back and forth to Python via NumPy arrays?

Maybe. I see that this is what Hans Petter does in a chapter of one of
his books. Still, there are issues with ownership.

Yes, we need to provide some way of passing a view of an other array, using
such a structure.

Post by Garth N. Wells
I guess in the end that memory management + pointers is always going to
be tough in a mixed-language environment.

For sure...

Johan

Post by Garth N. Wells
Garth

Post by Johan Hake
Including basic operations so it will be useful also in C++.
Johan

_______________________________________________
DOLFIN-dev mailing list
http://www.fenics.org/mailman/listinfo/dolfin-dev