OpenProject is the leading open source project management software.

kanbanworkflowstimelinescrumrubyroadmapproject-planningproject-managementopenprojectangularissue-trackerifcgantt-chartganttbug-trackerboardsbcf

You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

ulferts 3bf706b153 removes distinct as rails will handle it if necessary ... In case e.g. Projects are to be fetched via `Project.allowed_to?` rails will add the necessary distinct to the SQL statment on its own. But having the distinct forced in the query can lead to nonperformant subqueries and there, the distinct isn't necessary at all if the subquery is something like `where(project_id: Project.allowed(...))`. In such a case, whether the values are distinct will yield the same results. But the DB's query planner will take the distinct into account and the DB will have to work extra to remove the duplicates. In case of one query at least that extra work leads to an undesirable execution time of 8s compared to 60ms when avoiding the distinct. The executions plan with distinct is like this: ``` app=> EXPLAIN ANALYZE SELECT COUNT(*) FROM "notifications" WHERE "notifications"."read_ian" IS NOT NULL AND "notifications"."recipient_id" = 2147 AND "notifications"."resource_type" = 'WorkPackage' AND "notifications"."resource_id" IN (SELECT "work_packages"."id" FROM "work_packages" WHERE "work_packages"."project_id" IN (SELECT DISTINCT "projects"."id" FROM "projects" LEFT OUTER JOIN "members" ON "projects"."id" = "members"."project_id" AND "members"."user_id" = 2148 AND "projects"."active" = TRUE INNER JOIN "enabled_modules" ON "projects"."id" = "enabled_modules"."project_id" AND "enabled_modules"."name" IN ('work_package_tracking') AND "projects"."active" = TRUE INNER JOIN "role_permissions" ON "role_permissions"."permission" IN ('view_work_packages') INNER JOIN "roles" "permission_roles" ON "permission_roles"."id" = "role_permissions"."role_id" LEFT OUTER JOIN "member_roles" ON "members"."id" = "member_roles"."member_id" LEFT OUTER JOIN "roles" "assigned_roles" ON "assigned_roles"."id" = "permission_roles"."id" AND "projects"."active" = TRUE AND ("assigned_roles"."id" = "member_roles"."role_id" OR "projects"."public" = TRUE AND "assigned_roles"."builtin" = 1 AND "member_roles"."id" IS NULL) WHERE "assigned_roles"."id" IS NOT NULL)) AND (notifications.read_ian IN ('f')); QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Finalize Aggregate (cost=5060.65..5060.66 rows=1 width=8) (actual time=8060.041..8060.656 rows=1 loops=1) -> Gather (cost=5060.54..5060.65 rows=1 width=8) (actual time=8060.027..8060.652 rows=2 loops=1) Workers Planned: 1 Workers Launched: 1 -> Partial Aggregate (cost=4060.54..4060.55 rows=1 width=8) (actual time=8056.347..8056.357 rows=1 loops=2) -> Nested Loop Semi Join (cost=550.02..4059.44 rows=440 width=0) (actual time=83.255..8056.276 rows=62 loops=2) -> Parallel Bitmap Heap Scan on notifications (cost=103.77..1757.13 rows=1648 width=8) (actual time=0.461..7.019 rows=1826 loops=2) Recheck Cond: (recipient_id = 2147) Filter: ((read_ian IS NOT NULL) AND (NOT read_ian) AND ((resource_type)::text = 'WorkPackage'::text)) Heap Blocks: exact=523 -> Bitmap Index Scan on index_notifications_on_recipient_id (cost=0.00..103.07 rows=3554 width=0) (actual time=0.684..0.684 rows=3686 loops=1) Index Cond: (recipient_id = 2147) -> Hash Join (cost=446.24..447.36 rows=1 width=4) (actual time=4.398..4.398 rows=0 loops=3652) Hash Cond: (projects.id = work_packages.project_id) -> Unique (cost=441.49..441.78 rows=59 width=4) (actual time=4.357..4.373 rows=62 loops=3652) -> Sort (cost=441.49..441.64 rows=59 width=4) (actual time=4.357..4.362 rows=63 loops=3652) Sort Key: projects.id Sort Method: quicksort Memory: 28kB Worker 0: Sort Method: quicksort Memory: 28kB -> Merge Join (cost=159.35..439.75 rows=59 width=4) (actual time=0.499..4.343 rows=64 loops=3652) Merge Cond: (role_permissions.role_id = permission_roles.id) -> Nested Loop (cost=159.12..559.28 rows=89 width=12) (actual time=0.493..4.320 rows=64 loops=3652) Join Filter: ((assigned_roles.id = member_roles.role_id) OR (projects.public AND (assigned_roles.builtin = 1) AND (member_roles.id IS NULL))) Rows Removed by Join Filter: 21166 -> Merge Join (cost=82.31..84.06 rows=79 width=12) (actual time=0.419..0.491 rows=110 loops=3652) Merge Cond: (role_permissions.role_id = assigned_roles.id) -> Sort (cost=74.32..74.62 rows=120 width=4) (actual time=0.369..0.380 rows=120 loops=3652) Sort Key: role_permissions.role_id Sort Method: quicksort Memory: 30kB Worker 0: Sort Method: quicksort Memory: 30kB -> Seq Scan on role_permissions (cost=0.00..70.18 rows=120 width=4) (actual time=0.006..0.345 rows=121 loops=3652) Filter: ((permission)::text = 'view_work_packages'::text) Rows Removed by Filter: 3104 -> Sort (cost=7.98..8.27 rows=113 width=8) (actual time=0.048..0.057 rows=116 loops=3652) Sort Key: assigned_roles.id Sort Method: quicksort Memory: 30kB Worker 0: Sort Method: quicksort Memory: 30kB -> Seq Scan on roles assigned_roles (cost=0.00..4.13 rows=113 width=8) (actual time=0.004..0.027 rows=116 loops=3652) Filter: (id IS NOT NULL) -> Materialize (cost=76.81..299.96 rows=127 width=13) (actual time=0.000..0.013 rows=193 loops=401720) -> Hash Right Join (cost=76.81..299.32 rows=127 width=13) (actual time=0.622..0.946 rows=193 loops=2) Hash Cond: (members.project_id = projects.id) Join Filter: projects.active -> Nested Loop Left Join (cost=4.73..227.08 rows=25 width=12) (actual time=0.045..0.314 rows=68 loops=2) -> Bitmap Heap Scan on members (cost=4.44..69.05 rows=19 width=8) (actual time=0.028..0.060 rows=67 loops=2) Recheck Cond: (user_id = 2148) Heap Blocks: exact=12 -> Bitmap Index Scan on index_members_on_user_id_and_project_id (cost=0.00..4.43 rows=19 width=0) (actual time=0.019..0.020 rows=67 loops=2) Index Cond: (user_id = 2148) -> Index Scan using index_member_roles_on_member_id on member_roles (cost=0.29..8.31 rows=1 width=12) (actual time=0.003..0.003 rows=1 loops=134) Index Cond: (member_id = members.id) -> Hash (cost=70.89..70.89 rows=96 width=6) (actual time=0.558..0.562 rows=192 loops=2) Buckets: 1024 Batches: 1 Memory Usage: 16kB -> Hash Join (cost=53.25..70.89 rows=96 width=6) (actual time=0.296..0.510 rows=192 loops=2) Hash Cond: (enabled_modules.project_id = projects.id) -> Bitmap Heap Scan on enabled_modules (cost=15.16..31.81 rows=372 width=4) (actual time=0.064..0.184 rows=382 loops=2) Recheck Cond: ((name)::text = 'work_package_tracking'::text) Heap Blocks: exact=12 -> Bitmap Index Scan on index_enabled_modules_on_name (cost=0.00..15.07 rows=372 width=0) (actual time=0.054..0.055 rows=383 loops=2) Index Cond: ((name)::text = 'work_package_tracking'::text) -> Hash (cost=36.85..36.85 rows=99 width=6) (actual time=0.216..0.216 rows=195 loops=2) Buckets: 1024 Batches: 1 Memory Usage: 16kB -> Seq Scan on projects (cost=0.00..36.85 rows=99 width=6) (actual time=0.007..0.165 rows=195 loops=2) Filter: (active AND active) Rows Removed by Filter: 190 -> Index Only Scan using roles_pkey on roles permission_roles (cost=0.14..9.84 rows=113 width=4) (actual time=0.005..0.010 rows=33 loops=3652) Heap Fetches: 10956 -> Hash (cost=4.74..4.74 rows=1 width=8) (actual time=0.012..0.012 rows=1 loops=3652) Buckets: 1024 Batches: 1 Memory Usage: 9kB -> Index Scan using work_packages_pkey on work_packages (cost=0.42..4.74 rows=1 width=8) (actual time=0.009..0.010 rows=1 loops=3652) Index Cond: (id = notifications.resource_id) Planning Time: 2.242 ms Execution Time: 8060.808 ms ``` and without it: ``` EXPLAIN ANALYZE SELECT COUNT(*) FROM "notifications" WHERE "notifications"."read_ian" IS NOT NULL AND "notifications"."recipient_id" = 2147 AND "notifications"."resource_type" = 'WorkPackage' AND "notifications"."resource_id" IN (SELECT "work_packages"."id" FROM "work_packages" WHERE "work_packages"."project_id" IN (SELECT "projects"."id" FROM "projects" LEFT OUTER JOIN "members" ON "projects"."id" = "members"."project_id" AND "members"."user_id" = 2148 AND "projects"."active" = TRUE INNER JOIN "enabled_modules" ON "projects"."id" = "enabled_modules"."project_id" AND "enabled_modules"."name" IN ('work_package_tracking') AND "projects"."active" = TRUE INNER JOIN "role_permissions" ON "role_permissions"."permission" IN ('view_work_packages') INNER JOIN "roles" "permission_roles" ON "permission_roles"."id" = "role_permissions"."role_id" LEFT OUTER JOIN "member_roles" ON "members"."id" = "member_roles"."member_id" LEFT OUTER JOIN "roles" "assigned_roles" ON "assigned_roles"."id" = "permission_roles"."id" AND "projects"."active" = TRUE AND ("assigned_roles"."id" = "member_roles"."role_id" OR "projects"."public" = TRUE AND "assigned_roles"."builtin" = 1 AND "member_roles"."id" IS NULL) WHERE "assigned_roles"."id" IS NOT NULL)) AND (notifications.read_ian IN ('f')); QUERY PLAN -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Finalize Aggregate (cost=13644.10..13644.11 rows=1 width=8) (actual time=60.167..61.846 rows=1 loops=1) -> Gather (cost=13643.99..13644.10 rows=1 width=8) (actual time=60.154..61.839 rows=2 loops=1) Workers Planned: 1 Workers Launched: 1 -> Partial Aggregate (cost=12643.99..12644.00 rows=1 width=8) (actual time=56.361..56.376 rows=1 loops=2) -> Parallel Hash Semi Join (cost=10982.86..12643.36 rows=252 width=0) (actual time=53.275..56.364 rows=62 loops=2) Hash Cond: (notifications.resource_id = work_packages.id) -> Parallel Bitmap Heap Scan on notifications (cost=103.77..1757.13 rows=1648 width=8) (actual time=0.471..3.217 rows=1826 loops=2) Recheck Cond: (recipient_id = 2147) Filter: ((read_ian IS NOT NULL) AND (NOT read_ian) AND ((resource_type)::text = 'WorkPackage'::text)) Heap Blocks: exact=196 -> Bitmap Index Scan on index_notifications_on_recipient_id (cost=0.00..103.07 rows=3554 width=0) (actual time=0.712..0.712 rows=3686 loops=1) Index Cond: (recipient_id = 2147) -> Parallel Hash (cost=10782.56..10782.56 rows=7722 width=4) (actual time=50.813..50.826 rows=2328 loops=2) Buckets: 32768 Batches: 1 Memory Usage: 480kB -> Hash Semi Join (cost=440.49..10782.56 rows=7722 width=4) (actual time=9.231..49.907 rows=2328 loops=2) Hash Cond: (work_packages.project_id = projects.id) -> Parallel Seq Scan on work_packages (cost=0.00..10123.89 rows=50389 width=8) (actual time=0.006..16.405 rows=60496 loops=2) -> Hash (cost=439.75..439.75 rows=59 width=8) (actual time=9.196..9.208 rows=64 loops=2) Buckets: 1024 Batches: 1 Memory Usage: 11kB -> Merge Join (cost=159.35..439.75 rows=59 width=8) (actual time=2.212..9.188 rows=64 loops=2) Merge Cond: (role_permissions.role_id = permission_roles.id) -> Nested Loop (cost=159.12..559.28 rows=89 width=16) (actual time=2.185..9.141 rows=64 loops=2) Join Filter: ((assigned_roles.id = member_roles.role_id) OR (projects.public AND (assigned_roles.builtin = 1) AND (member_roles.id IS NULL))) Rows Removed by Join Filter: 21166 -> Merge Join (cost=82.31..84.06 rows=79 width=12) (actual time=1.272..1.360 rows=110 loops=2) Merge Cond: (role_permissions.role_id = assigned_roles.id) -> Sort (cost=74.32..74.62 rows=120 width=4) (actual time=1.167..1.183 rows=120 loops=2) Sort Key: role_permissions.role_id Sort Method: quicksort Memory: 30kB Worker 0: Sort Method: quicksort Memory: 30kB -> Seq Scan on role_permissions (cost=0.00..70.18 rows=120 width=4) (actual time=0.015..1.121 rows=121 loops=2) Filter: ((permission)::text = 'view_work_packages'::text) Rows Removed by Filter: 3104 -> Sort (cost=7.98..8.27 rows=113 width=8) (actual time=0.102..0.116 rows=116 loops=2) Sort Key: assigned_roles.id Sort Method: quicksort Memory: 30kB Worker 0: Sort Method: quicksort Memory: 30kB -> Seq Scan on roles assigned_roles (cost=0.00..4.13 rows=113 width=8) (actual time=0.015..0.044 rows=116 loops=2) Filter: (id IS NOT NULL) -> Materialize (cost=76.81..299.96 rows=127 width=17) (actual time=0.005..0.039 rows=193 loops=220) -> Hash Right Join (cost=76.81..299.32 rows=127 width=17) (actual time=0.548..0.776 rows=193 loops=2) Hash Cond: (members.project_id = projects.id) Join Filter: projects.active -> Nested Loop Left Join (cost=4.73..227.08 rows=25 width=12) (actual time=0.046..0.218 rows=68 loops=2) -> Bitmap Heap Scan on members (cost=4.44..69.05 rows=19 width=8) (actual time=0.027..0.056 rows=67 loops=2) Recheck Cond: (user_id = 2148) Heap Blocks: exact=12 -> Bitmap Index Scan on index_members_on_user_id_and_project_id (cost=0.00..4.43 rows=19 width=0) (actual time=0.021..0.021 rows=67 loops=2) Index Cond: (user_id = 2148) -> Index Scan using index_member_roles_on_member_id on member_roles (cost=0.29..8.31 rows=1 width=12) (actual time=0.002..0.002 rows=1 loops=134) Index Cond: (member_id = members.id) -> Hash (cost=70.89..70.89 rows=96 width=10) (actual time=0.488..0.491 rows=192 loops=2) Buckets: 1024 Batches: 1 Memory Usage: 17kB -> Hash Join (cost=53.25..70.89 rows=96 width=10) (actual time=0.277..0.443 rows=192 loops=2) Hash Cond: (enabled_modules.project_id = projects.id) -> Bitmap Heap Scan on enabled_modules (cost=15.16..31.81 rows=372 width=4) (actual time=0.061..0.137 rows=382 loops=2) Recheck Cond: ((name)::text = 'work_package_tracking'::text) Heap Blocks: exact=12 -> Bitmap Index Scan on index_enabled_modules_on_name (cost=0.00..15.07 rows=372 width=0) (actual time=0.052..0.052 rows=383 loops=2) Index Cond: ((name)::text = 'work_package_tracking'::text) -> Hash (cost=36.85..36.85 rows=99 width=6) (actual time=0.200..0.201 rows=195 loops=2) Buckets: 1024 Batches: 1 Memory Usage: 16kB -> Seq Scan on projects (cost=0.00..36.85 rows=99 width=6) (actual time=0.009..0.155 rows=195 loops=2) Filter: (active AND active) Rows Removed by Filter: 190 -> Index Only Scan using roles_pkey on roles permission_roles (cost=0.14..9.84 rows=113 width=4) (actual time=0.023..0.030 rows=33 loops=2) Heap Fetches: 6 Planning Time: 2.247 ms Execution Time: 61.963 ms ``` The key difference is in the number of loops in the first statement.		3 years ago
..
abstract_query.rb	removes distinct as rails will handle it if necessary	3 years ago
abstract_user_query.rb	remove ruby magic comment for utf-8	3 years ago
enterprise_service.rb	remove ruby magic comment for utf-8	3 years ago
project_query.rb	remove ruby magic comment for utf-8	3 years ago
query_transformation.rb	remove ruby magic comment for utf-8	3 years ago
query_transformation_visitor.rb	remove ruby magic comment for utf-8	3 years ago
query_transformations.rb	remove ruby magic comment for utf-8	3 years ago
query_transformations_order.rb	remove ruby magic comment for utf-8	3 years ago
user_allowed_query.rb	remove ruby magic comment for utf-8	3 years ago
user_allowed_service.rb	remove ruby magic comment for utf-8	3 years ago
user_global_roles_query.rb	remove ruby magic comment for utf-8	3 years ago
user_project_roles_query.rb	remove ruby magic comment for utf-8	3 years ago
user_roles_query.rb	remove ruby magic comment for utf-8	3 years ago