Add a value to a Redis list only if it doesn't already exist in the list?

I'm trying to add a value to a list but only if it hasn't been added yet.

Is there a command to do this or is there a way to test for the existence of a value within a list?

Thanks!

It looks like you need a set or a sorted set.

Sets have O(1) membership test and enforced uniqueness.

Such feature is available in set using `hexists`[hexists](

[To see links please register here]

) command in redis.

Checking a list to see if a member exists within it is O(n), which can get quite expensive for big lists and is definitely not ideal. That said, everyone else seems to be giving you alternatives. I'll just tell you how to do what you're asking to do, and assume you have good reasons for doing it the way you're doing it. I'll do it in Python, assuming you have a connection to Redis called `r`, some list called `some_list` and some new item to add called `new_item`:

lst = r.lrange(list_name, -float('Inf'), float('Inf'))
if new_item not in lst:
r.rpush(list_name, new_item)

I need to do the same.
I think about to remove the element from the list and then add it again. If the element is not in the list, redis will return 0, so there is no error

lrem mylist 0 myitem
rpush mylist myitem

As Tommaso Barbugli mentioned you should use a set instead a list if you need only unique values.
[see REDIS documentation SADD][1]

redis> SADD myset "Hello"
(integer) 1
redis> SADD myset "World"
(integer) 1
redis> SADD myset "World"
(integer) 0
redis> SMEMBERS myset
1) "World"
2) "Hello"

If you want to check the presence of a value in the set you may use [SISMEMBER][2]

redis> SADD myset "one"
(integer) 1
redis> SISMEMBER myset "one"
(integer) 1
redis> SISMEMBER myset "two"
(integer) 0



[1]:

[To see links please register here]

[2]:

[To see links please register here]

I encountered this problem while adding to a task worker queue, because I wanted to avoid adding many duplicate tasks. Using a Redis set (as many people are suggesting) would be nice, but Redis sets don't have a "blocking pop" like BRPOPLPUSH, so they're not good for task queues.

So, here's my slightly non-ideal solution (in Python):

def pushOnlyNewItemsToList(redis, list_name, items):
""" Adds only the items that aren't already in the list.
Though if run simultaneously in multiple threads, there's still a tiny chance of adding duplicate items.
O(n) on the size of the list."""
existing_items = set(redis.lrange(list_name,0,-1))
new_items = set(items).difference(existing_items)
if new_items:
redis.lpush(list_name, *new_items)

Note the caveats in the docstring.

If you need to truly guarantee no duplicates, the alternative is to run LREM, LPUSH inside a Redis pipeline, as in 0xAffe's answer. That approach causes less network traffic, but has the downside of reordering the list. It's probably the best general answer if you don't care about list order.

If you can't use the SETs (in case you want to achieve some blocking POP/PUSH list features) you can use a simple script:

script load 'local exists = false; for idx=1, redis.call("LLEN",KEYS[1]) do if (redis.call("LINDEX", KEYS[1], idx) == ARGV[1]) then exists = true; break; end end; if (not exists) then redis.call("RPUSH", KEYS[1], ARGV[1]) end; return not exists or 0'

This will return the SHA code of the script you've added.

Just call then:

evalsha 3e31bb17571f819bea95ca5eb5747a373c575ad9 1 test-list myval

where

- `3e31bb17571f819bea95ca5eb5747a373c575ad9` (the SHA code of the script you added)
- `1` — is number of parameters (1 is constant for this function)
- `test-list` — the name of your list
- `myval` - the value you need to add

it returns 1 if the new item was added or 0 if it was already in the list.

As @Eli states, checking for existence in a list is a O(N) operation, which is costly especially if the list is large. I'm faced with a similar problem. For me using a SET is not an option because I need the insert ordered to be guaranteed when removing items from the list. Redis SETS remove/retrieve items in random order, which is a deal breaker for me. What I'm thinking of doing is maintaining a separate HASH in Redis just to check if the item exists already exists before adding it to the list. This means that whenever I add an item in the list, I have to add it to this secondary data structure (the HASH), to support a O(1) lookup to check for existence. I know this is duplicating data, but given the other options the additional space consumption might not seem so bad.