(callbacks and passes)
Interpreter for GIMPLE (simple form of abstract interpretation)
See libcpychecker/absinterp.py in the plugin's source tree
We generate a set of all possible traces through the function
(but give up when it gets too complicated)
A trace is a recording of a list of states, with transitions between them.
Transitions can have descriptive information (e.g. "when PyList_Append() fails")
A state has:
- location
- mapping from variables to storage region
- mapping from storage region to AbstractValue instances
States are pluggable: extra domain-specific information ("facets") can be added.
For now, the only plugin is a "cpython" one which adds:
- exception state (the PyObject* for the thread-local exception instance, or NULL)
- GIL state
- detailed behavior information on much of the CPython API
See libcpychecker/refcounts.py in the plugin's source tree
Given a C function e.g. PyMapping_Size, the cpython facet can provide an implementation which describes how to get from an input state to a list of all possible transitions:
def impl_PyMapping_Size(self, stmt, v_o):
fnmeta = FnMeta(name='PyMapping_Size',
docurl='http://docs.python.org/c-api/mapping.html#PyMapping_Size',
prototype='Py_ssize_t PyMapping_Size(PyObject *o)',
defined_in='Objects/abstract.c',
notes='Can cope with NULL (sets exception)')
t_success = self.state.mktrans_assignment(stmt.lhs,
UnknownValue.make(stmt.lhs.type,
stmt.loc),
fnmeta.desc_when_call_succeeds())
t_failure = self.state.mktrans_assignment(stmt.lhs,
ConcreteValue(stmt.lhs.type,
stmt.loc,
-1),
fnmeta.desc_when_call_fails())
t_failure.dest.cpython.set_exception('PyExc_TypeError',
stmt.loc)
return [t_success, t_failure]
It only traces zero or one times any loop; it gives up after that.
l-values and r-values
It emits new compilation warnings:
$ ./gcc-with-cpychecker \
-I/usr/include/python2.7/ bug.c
bug.c: In function ‘make_a_list_of_random_ints_badly’:
bug.c:10:26: warning: Mismatching type in
call to PyArg_ParseTuple with format code "i" [enabled
by default]
argument 3 ("&count") had type
"long int *" (pointing to 64 bits)
but was expecting
"int *" (pointing to 32 bits)
for format code "i"
bug.c:22:1: warning: ob_refcnt of '*item' is
1 too high [enabled by default]
bug.c:22:1: note: was expecting final
ob_refcnt to be N + 1 (for some unknown N)
bug.c:22:1: note: due to object being
referenced by: PyListObject.ob_item[0]
bug.c:22:1: note: but final ob_refcnt is
N + 2
bug.c:17:14: note: PyLongObject allocated
at:
item = PyLong_FromLong(random());
For every object it tracks all reference that it knows about, and assumes that there is some unknown number of other references N that it isn't tracking.
It knows the lower limit of N, and can use this when analyzing Py_DECREF() invocations.
FIXME
The reference count warnings emits a verbose description of the path followed through the code that leads to the error:
bug.c:10:26: note: when PyArg_ParseTuple()
succeeds at:
if (!PyArg_ParseTuple(args, "i", &count)) {
bug.c:10:8: note: taking False path at:
if (!PyArg_ParseTuple(args, "i", &count)) {
bug.c:14:10: note: reaching:
list = PyList_New(0);
bug.c:14:10: note: when PyList_New()
succeeds at:
list = PyList_New(0);
bug.c:16:5: note: when considering range:
1 <= value <= 0x7fffffff at:
for (i = 0; i < count; i++) {
bug.c:16:5: note: taking True path at:
for (i = 0; i < count; i++) {
bug.c:17:31: note: reaching:
item = PyLong_FromLong(random());
The checker emits a message when either changes:
bug.c:17:14: note: when PyLong_FromLong() succeeds at: item = PyLong_FromLong(random()); bug.c:17:14: note: ob_refcnt is now refs: 1 + N where N >= 0 bug.c:18:22: note: when PyList_Append() succeeds at: PyList_Append(list, item); bug.c:18:22: note: ob_refcnt is now refs: 2 + N where N >= 0 bug.c:18:22: note: '*item' is now referenced by 1 non-stack value(s): PyListObject.ob_item[0]
The checker continues following the execution of the code, reaching the top of the loop:
bug.c:16:5: note: when considering
value == (int)1 from bug.c:10 at:
for (i = 0; i < count; i++) {
bug.c:16:5: note: taking False path at:
for (i = 0; i < count; i++) {
bug.c:21:5: note: reaching:
return list;
bug.c:22:1: note: returning
The checker attempts to follow every path through the function.
In this case it detects two similar traces, and only reports the warning for the first such trace:
bug.c:22:1: note: found 1 similar trace(s) to this
The above information is too verbose, so the checker also emits the same information in HTML form:
bug.c:6:1: note: graphical error report for function 'make_a_list_of_random_ints_badly' written out to 'bug.c.make_a_list_of_random_ints_badly -refcount-errors.html'
FIXME: screenshot
It also detects the lack of error-checking for the PyList_New() call:
bug.c:18:22: warning: calling PyList_Append with NULL
as argument 1 (list) at bug.c:18 [enabled by default]
bug.c:10:26: note: when PyArg_ParseTuple() succeeds at:
if (!PyArg_ParseTuple(args, "i", &count)) {
bug.c:10:8: note: taking False path at:
if (!PyArg_ParseTuple(args, "i", &count)) {
bug.c:14:10: note: reaching:
list = PyList_New(0);
bug.c:14:10: note: when PyList_New() fails at:
list = PyList_New(0);
bug.c:16:5: note: when considering range:
1 <= value <= 0x7fffffff at:
for (i = 0; i < count; i++) {
bug.c:16:5: note: taking True path at:
for (i = 0; i < count; i++) {
bug.c:17:31: note: reaching:
item = PyLong_FromLong(random());
bug.c:17:14: note: when PyLong_FromLong() succeeds at:
item = PyLong_FromLong(random());
bug.c:18:22: note: PyList_Append() invokes Py_TYPE() on
the pointer via the PyList_Check() macro, thus accessing
(NULL)->ob_type
bug.c:18:22: note: found 1 similar trace(s) to this
bug.c:6:1: note: graphical error report for function 'make_a_list_of_random_ints_badly' written out to 'bug.c.make_a_list_of_random_ints_badly-refcount-errors.html'
It only complains about crashers and reference-count issues
It doesn't complain about the lack of error checking for PyLong_FromLong: this failure case can't crash the interpreter.
some of the reference-count bugs found (and fixed) with the tool (and other bugs)