[PonyORM-list] Pony's Cache Size Limit?

[Matthew Bell]

You were right, my code was lazily thought out. With one query (with one
subquery), the execution is fast and memory consumption normal.

Thanks!

On 8 November 2014 13:19, Alexander Kozlovsky <alexander.kozlovsky at gmail.com
> wrote:

> Hi, Matthew
>
> If you retrieve these objects to render some HTML page, than I doubt that
> you really need so many objects. I think that a typical web page requires
> several thousands objects at most, and this amount is not big enough to
> cause memory problems.
>
> > each time the inner function is called, it returns a data structure
> full of pony objects (but a lot less pony objects than the function
> internally used)
>
> Ideally you function should retrieve only the objects which are actually
> required to render the Jinja template. If the function retrieves much more
> objects, but then uses only small fraction of them, then it probably
> written inefficiently. You should do object filtering not in memory, but in
> the initial database query.
>
> If you can show me the database query then I probably can suggest a better
> way to do filtering, so only required objects will be materialized in
> memory.
>
> Regards,
> Alexander
>
>
> On Fri, Nov 7, 2014 at 5:40 PM, Matthew Bell <matthewrobertbell at gmail.com>
> wrote:
>
>> I think the problem is that each time the inner function is called, it
>> returns a data  structure full of pony objects (but a lot less pony objects
>> than the function internally used). I could pull the attributes of the
>> objects that I need (to inject into a Jinja template, but not for a
>> webapp), but that seems hacky. I could pickle / just return the IDs of the
>> objects, but that also seems hacky.
>>
>> The relevant line in the jinja template is like:
>>
>> {% for keyword in keywords %}
>>    {{ keyword.expensive_method() }}
>>
>> Do you see any solutions?
>>
>> Thanks
>>
>> On 6 November 2014 13:05, Alexander Kozlovsky <
>> alexander.kozlovsky at gmail.com> wrote:
>>
>>> Hi Matthew,
>>>
>>> > Does Pony's in-memory cache have a size limit?
>>>
>>> Currently no, but we can think about it. It may be not so easy to
>>> implement, because objects in cache have cross-relations.
>>> May be we can add some method like db_session.forget(MyEntity) to clear
>>> all unmodified entities of this type from the cache. Can this help in your
>>> case?
>>>
>>> >  I have a function which does a lot of queries within one transaction
>>>
>>> Are you sure that in your case all those queries must be done in the
>>> same transaction? If not, then wrap different queries in different
>>> db_sessions, and old objects should be cleared automatically.
>>>
>>>
>>> On Thu, Nov 6, 2014 at 3:43 PM, Matthew Bell <
>>> matthewrobertbell at gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Does Pony's in-memory cache have a size limit? I have a function which
>>>> does a lot of queries within one transaction, dereferencing > 99% of the
>>>> pony objects, but the memory used by the process grows and grows and the
>>>> function goes through its loops.
>>>>
>>>> Psuedo code:
>>>>
>>>> results = []
>>>> for i in range(5):
>>>>   results.append(some_function_which_queries_lots(i))
>>>>
>>>> Thanks!
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Matthew Bell
>>>>
>>>> _______________________________________________
>>>> ponyorm-list mailing list
>>>> ponyorm-list at ponyorm.org
>>>> /ponyorm-list
>>>>
>>>>
>>>
>>> _______________________________________________
>>> ponyorm-list mailing list
>>> ponyorm-list at ponyorm.org
>>> /ponyorm-list
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Matthew Bell
>>
>> _______________________________________________
>> ponyorm-list mailing list
>> ponyorm-list at ponyorm.org
>> /ponyorm-list
>>
>>
>
> _______________________________________________
> ponyorm-list mailing list
> ponyorm-list at ponyorm.org
> /ponyorm-list
>
>


-- 
Regards,

Matthew Bell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: </ponyorm-list/attachments/20141109/ea027e9e/attachment.html>